DaddyFrosty/yoga
1
0
Fork 0
Code Issues 96 Pull Requests 31 Actions Packages Projects Releases 33 Wiki Activity

Add simple fuzz-harness #1537

Closed
nathaniel-brough wants to merge 2 commits from add-fuzz-harness into main
pull from: add-fuzz-harness
merge into: DaddyFrosty:main
Conversation 17 Commits 2 Files Changed 5 +120
nathaniel-brough commented 2024-01-08 12:34:09 -08:00 (Migrated from github.com)
Copy Link

This PR is based on the proposal in https://github.com/facebook/yoga/issues/1538 inlined below, to integrate yoga with oss-fuzz

Hey yoga team,

I've recently become interested in yoga. I'd like to suggest and champion an effort to set up some basic fuzz-testing and combine it with google/oss-fuzz for continuous fuzzing. I'm fully aware that you are very busy people and I don't want to overload your review/maintenance capacity. Is this a bad time to discuss potential security/reliability improvements?

If you're not familiar with fuzzing or oss-fuzz I've included a few brief notes below.

Benefits of Fuzz-Testing

  • Dynamic Code Testing: Fuzz-testing challenges systems with unexpected data, aiming to identify vulnerabilities or bugs. It’s akin to an exhaustive stress-test for the code.
  • Detecting Hidden Vulnerabilities: It can uncover potential weaknesses that may not be evident in routine tests.
  • Continuous and Automated Testing: With tools like Google’s OSS-Fuzz, fuzz-testing can be automated, running continuously on distributed systems, ensuring daily resilience checks.

Google/oss-fuzz for Continuous Fuzzing

  • Automated Fuzzing: OSS-Fuzz undertakes comprehensive fuzz-testing daily on a distributed cluster.
  • Detailed Reporting: OSS-Fuzz offers exhaustive reports in case of detected anomalies, enabling effective action.

I’d be more than happy to lead the effort in integrating fuzz testing with the yoga and assist in any way required.

Prior integrations

There have been a number of previous integrations completed with facebook repositories and google/oss-fuzz including;

  • facebook/time
  • facebook/zstd
  • facebookexperimental/starlark-rust (this was me)
  • facebook/proxygen
  • facebook/hermes
  • facebook/rocksdb

As a proof of concept I created a couple of super simple fuzz harnesses in #1537.

NOTE: Adding fuzz-testing and integrating with google/oss-fuzz was previously suggested here https://github.com/facebook/yoga/pull/1055 and was rejected. I think I've addressed the concerns raised in the first PR. While the original PR contained what was probably a higher performance fuzzer, the new fuzzer should be easier to integrate and doesn't introduce multiple sources of truth.

> This PR is based on the proposal in https://github.com/facebook/yoga/issues/1538 inlined below, to integrate yoga with `oss-fuzz` Hey yoga team, I've recently become interested in yoga. I'd like to suggest and champion an effort to set up some basic fuzz-testing and combine it with google/oss-fuzz for continuous fuzzing. I'm fully aware that you are very busy people and I don't want to overload your review/maintenance capacity. Is this a bad time to discuss potential security/reliability improvements? If you're not familiar with fuzzing or oss-fuzz I've included a few brief notes below. #### **Benefits of Fuzz-Testing** - **Dynamic Code Testing**: Fuzz-testing challenges systems with unexpected data, aiming to identify vulnerabilities or bugs. It’s akin to an exhaustive stress-test for the code. - **Detecting Hidden Vulnerabilities**: It can uncover potential weaknesses that may not be evident in routine tests. - **Continuous and Automated Testing**: With tools like Google’s OSS-Fuzz, fuzz-testing can be automated, running continuously on distributed systems, ensuring daily resilience checks. #### **Google/oss-fuzz for Continuous Fuzzing** - **Automated Fuzzing**: OSS-Fuzz undertakes comprehensive fuzz-testing daily on a distributed cluster. - **Detailed Reporting**: OSS-Fuzz offers exhaustive reports in case of detected anomalies, enabling effective action. I’d be more than happy to lead the effort in integrating fuzz testing with the yoga and assist in any way required. #### Prior integrations There have been a number of previous integrations completed with facebook repositories and google/oss-fuzz including; - facebook/time - facebook/zstd - facebookexperimental/starlark-rust (this was me) - facebook/proxygen - facebook/hermes - facebook/rocksdb As a proof of concept I created a couple of super simple fuzz harnesses in #1537. NOTE: Adding fuzz-testing and integrating with google/oss-fuzz was previously suggested here https://github.com/facebook/yoga/pull/1055 and was rejected. I think I've addressed the concerns raised in the first PR. While the original PR contained what was probably a higher performance fuzzer, the new fuzzer should be easier to integrate and doesn't introduce multiple sources of truth.
vercel[bot] commented 2024-01-08 12:34:14 -08:00 (Migrated from github.com)
Copy Link

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment
Name Status Preview Comments Updated (UTC)
yoga-website-next ⬜️ Ignored (Inspect) Visit Preview Jan 13, 2024 1:38am
[vc]: #m//pnYn84t6IY5+ECT0M/XMC4uLVEOfeUyDmnNfsxGo=:eyJpc01vbm9yZXBvIjp0cnVlLCJ0eXBlIjoiZ2l0aHViIiwicHJvamVjdHMiOlt7Im5hbWUiOiJ5b2dhLXdlYnNpdGUtbmV4dCIsInJvb3REaXJlY3RvcnkiOiJ3ZWJzaXRlLW5leHQiLCJpbnNwZWN0b3JVcmwiOiJodHRwczovL3ZlcmNlbC5jb20vZmJvcGVuc291cmNlL3lvZ2Etd2Vic2l0ZS1uZXh0LzV3eFVvOWlMbWtwZ21vcHNxSnlpN3AxYjRKdlYiLCJwcmV2aWV3VXJsIjoieW9nYS13ZWJzaXRlLW5leHQtZ2l0LWZvcmstc2lsdmVyZ2FzcC1hZGQtZi03YWI4YmQtZmJvcGVuc291cmNlLnZlcmNlbC5hcHAiLCJuZXh0Q29tbWl0U3RhdHVzIjoiSUdOT1JFRCIsImxpdmVGZWVkYmFjayI6eyJyZXNvbHZlZCI6MCwidW5yZXNvbHZlZCI6MCwidG90YWwiOjAsImxpbmsiOiIifX1dfQ== **The latest updates on your projects**. Learn more about [Vercel for Git ↗︎](https://vercel.link/github-learn-more) <details><summary>1 Ignored Deployment</summary> | Name | Status | Preview | Comments | Updated (UTC) | | :--- | :----- | :------ | :------- | :------ | | **yoga-website-next** | ⬜️ Ignored ([Inspect](https://vercel.com/fbopensource/yoga-website-next/5wxUo9iLmkpgmopsqJyi7p1b4JvV)) | [Visit Preview](https://yoga-website-next-git-fork-silvergasp-add-f-7ab8bd-fbopensource.vercel.app) | | Jan 13, 2024 1:38am | </details>
NickGerleman (Migrated from github.com) requested changes 2024-01-09 18:39:45 -08:00
NickGerleman (Migrated from github.com) left a comment
Copy Link

Thanks for the contribution! I read through https://github.com/facebook/yoga/issues/1538 and looked through this code, and can definitely see the value here.

My main ask, if that if we want code which fuzzes Yoga to live in the Yoga repo, the Yoga repo should also have automation to build (and ideally run) the fuzzer. E.g. add GitHubs actions workflows exercising what this PR adds.

Otherwise, the changes here look good. If you could inline your proosal, or link to it, in the PR body, that would be helpful when looking back at commits.

Thanks for the contribution! I read through https://github.com/facebook/yoga/issues/1538 and looked through this code, and can definitely see the value here. My main ask, if that if we want code which fuzzes Yoga to live in the Yoga repo, the Yoga repo should also have automation to build (and ideally run) the fuzzer. E.g. add GitHubs actions workflows exercising what this PR adds. Otherwise, the changes here look good. If you could inline your proosal, or link to it, in the PR body, that would be helpful when looking back at commits.
NickGerleman commented 2024-01-09 18:52:40 -08:00 (Migrated from github.com)
Copy Link

I read more about google/oss-fuzz, and see that they would be running the fuzzer continuously.

So, I think we would just want some build automation for this, so we don't accidentally break it without noticing. I think that would look like adding a job to https://github.com/facebook/yoga/blob/main/.github/workflows/validate-cpp.yml

I read more about `google/oss-fuzz`, and see that they would be running the fuzzer continuously. So, I think we would just want some build automation for this, so we don't accidentally break it without noticing. I think that would look like adding a job to https://github.com/facebook/yoga/blob/main/.github/workflows/validate-cpp.yml
nathaniel-brough commented 2024-01-10 09:42:26 -08:00 (Migrated from github.com)
Copy Link

Looks like I need to fix some formatting.

I read more about google/oss-fuzz, and see that they would be running the fuzzer continuously.
So, I think we would just want some build automation for this, so we don't accidentally break it without noticing. I think that would look like adding a job to https://github.com/facebook/yoga/blob/main/.github/workflows/validate-cpp.yml

There is an oss-fuzz specific github action, which can be useful for this sort of thing. It'll build/run the fuzzers for a short period on every pull request. The goal being to catch shallow bugs before they merge, with the additional benefit that it'll prevent regressions as it shares the corpus from the main oss-fuzz cluster.

Looks like I need to fix some formatting. > I read more about `google/oss-fuzz`, and see that they would be running the fuzzer continuously. > So, I think we would just want some build automation for this, so we don't accidentally break it without noticing. I think that would look like adding a job to https://github.com/facebook/yoga/blob/main/.github/workflows/validate-cpp.yml There is an oss-fuzz specific [github action](https://google.github.io/oss-fuzz/getting-started/continuous-integration/), which can be useful for this sort of thing. It'll build/run the fuzzers for a short period on every pull request. The goal being to catch shallow bugs before they merge, with the additional benefit that it'll prevent regressions as it shares the corpus from the main oss-fuzz cluster.
nathaniel-brough commented 2024-01-10 12:56:35 -08:00 (Migrated from github.com)
Copy Link

I've gone with a simple CI script as you've suggested for now, we can look at using the oss-fuzz github action in future.

I've gone with a simple CI script as you've suggested for now, we can look at using the oss-fuzz github action in future.
NickGerleman commented 2024-01-10 23:30:52 -08:00 (Migrated from github.com)
Copy Link

@silvergasp I hit the button to run, and looks like the action currently runs into a linker error.

@silvergasp I hit the button to run, and looks like the action currently runs into a linker error.
nathaniel-brough commented 2024-01-11 07:07:40 -08:00 (Migrated from github.com)
Copy Link

@silvergasp I hit the button to run, and looks like the action currently runs into a linker error.

Hmm that's odd I was just running the fuzzer on my machine yesterday. I'll experiment a bit with the CI on my branch so that I can hit the CI button myself. I'll ping you again here once I've fixed the problem.

> @silvergasp I hit the button to run, and looks like the action currently runs into a linker error. Hmm that's odd I was just running the fuzzer on my machine yesterday. I'll experiment a bit with the CI on my branch so that I can hit the CI button myself. I'll ping you again here once I've fixed the problem.
nathaniel-brough commented 2024-01-12 17:48:42 -08:00 (Migrated from github.com)
Copy Link

Ok it took a little fiddling but I think I've got the incantation correct now. The issue was the the compiler runtime that the fuzzer uses is (at least on ubuntu) linked against libstdc++ but yoga is linked against libc++ on the CI, this lead to some conflicts and missing symbols when linking. The things I tried;

  • Switching everything over to use stdc++ instead of libc++. This lead to compiler errors in gtest.
  • Building libfuzzer again from source and linking against libc++ instead of libstdc++. This isn't difficult, but adds build time and complexity unnecessarily.
  • Switching over to stdlibc++ and then only building the fuzzer's not any of the unit tests. This is the solution that I settled on.

TLDR; this is green now :)

Ok it took a little fiddling but I think I've got the incantation correct now. The issue was the the compiler runtime that the fuzzer uses is (at least on ubuntu) linked against libstdc++ but yoga is linked against [libc++](https://github.com/facebook/yoga/blob/8c3e01166bc5b6c9e100ef95b2f29f8bc3fdd98f/.github/actions/setup-cpp/action.yml#L22) on the CI, this lead to some conflicts and missing symbols when linking. The things I tried; - Switching everything over to use stdc++ instead of libc++. This lead to [compiler errors in gtest](https://github.com/silvergasp/yoga/actions/runs/7509139991/job/20445818849#step:4:114). - Building libfuzzer again from source and linking against libc++ instead of libstdc++. This isn't difficult, but adds build time and complexity unnecessarily. - Switching over to stdlibc++ and then only building the fuzzer's not any of the unit tests. This is the solution that I settled on. TLDR; this is green now :)
NickGerleman commented 2024-01-13 02:05:22 -08:00 (Migrated from github.com)
Copy Link

Thanks for the explanation.

In 6a72a7a20b/.github/actions/setup-cpp/action.yml we usually either setup the machines as gcc and stdlib++ or clang and libc++.

Does this work if we keep to that pattern, and use GCC as OS default? Or does it need to be Clang?

Thanks for the explanation. In https://github.com/facebook/yoga/blob/6a72a7a20b05cf646ae43d63994c9f6dbb5c123c/.github/actions/setup-cpp/action.yml we usually either setup the machines as gcc and stdlib++ or clang and libc++. Does this work if we keep to that pattern, and use GCC as OS default? Or does it need to be Clang?
nathaniel-brough commented 2024-01-13 09:58:53 -08:00 (Migrated from github.com)
Copy Link

Thanks for the explanation.

In 6a72a7a20b/.github/actions/setup-cpp/action.yml we usually either setup the machines as gcc and stdlib++ or clang and libc++.

Hmm where is the stdlib++ build setup, I wasn't able to get a working complete build using stdlib++, i.e. without gtest compiler failures.

Does this work if we keep to that pattern, and use GCC as OS default? Or does it need to be Clang?

As far as I'm aware libfuzzer only works with clang, as it is an llvm project. There are other fuzzing engines that support gcc e.g. AFL. But google/oss-fuzz essentially provides adapters for libfuzzer based fuzzers and it'll link in AFL/Centifpede/honggfuzz under the same interface of libfuzzer. As far as I'm aware oss-fuzz only uses clang.

> Thanks for the explanation. > > In https://github.com/facebook/yoga/blob/6a72a7a20b05cf646ae43d63994c9f6dbb5c123c/.github/actions/setup-cpp/action.yml we usually either setup the machines as gcc and stdlib++ or clang and libc++. Hmm where is the stdlib++ build setup, I wasn't able to get a working complete build using stdlib++, i.e. without gtest compiler failures. > Does this work if we keep to that pattern, and use GCC as OS default? Or does it need to be Clang? As far as I'm aware [libfuzzer](https://www.llvm.org/docs/LibFuzzer.html) only works with clang, as it is an llvm project. There are other fuzzing engines that support gcc e.g. AFL. But google/oss-fuzz essentially provides adapters for libfuzzer based fuzzers and it'll link in AFL/Centifpede/honggfuzz under the same interface of libfuzzer. As far as I'm aware oss-fuzz only uses clang.
NickGerleman (Migrated from github.com) approved these changes 2024-01-16 06:00:10 -08:00
NickGerleman commented 2024-01-16 06:06:47 -08:00 (Migrated from github.com)
Copy Link

Ah, thanks for the context.

Hmm where is the stdlib++ build setup, I wasn't able to get a working complete build using stdlib++, i.e. without gtest compiler failures.

We use stdlibc++ alongside the GCC reference build, but set Clang build to use libc++, since it is more often tested with it. As you noticed, this can conflict with any libraries installed by the system (on Linux).

That change was actually made, when Ubuntu LTS bumped some version of Clang or libstdc++, so that importing the chrono header under C++ 20 was enough to produce a compile error. Ubuntu LTS is using some older compilers/libraries where C++ 20 was not as mature, but it seems like they don't really try hard to support the combination (e.g. why we see gtest failure).

Fuzz here runs into that kinda unsupported state right now, but it does build, and might be less error-prone if we use OSS fuzz action to build outside this infra.

Ah, thanks for the context. > Hmm where is the stdlib++ build setup, I wasn't able to get a working complete build using stdlib++, i.e. without gtest compiler failures. We use `stdlibc++` alongside the GCC reference build, but set Clang build to use `libc++`, since it is more often tested with it. As you noticed, this can conflict with any libraries installed by the system (on Linux). That change was actually made, when Ubuntu LTS bumped some version of Clang or `libstdc++`, so that importing the `chrono` header under C++ 20 was enough to produce a compile error. Ubuntu LTS is using some older compilers/libraries where C++ 20 was not as mature, but it seems like they don't really try hard to support the combination (e.g. why we see gtest failure). Fuzz here runs into that kinda unsupported state right now, but it does build, and might be less error-prone if we use OSS fuzz action to build outside this infra.
👍 1
facebook-github-bot commented 2024-01-16 06:07:26 -08:00 (Migrated from github.com)
Copy Link

@NickGerleman has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@NickGerleman has imported this pull request. If you are a Meta employee, you can view this diff [on Phabricator](https://www.internalfb.com/diff/D52800366).
NickGerleman (Migrated from github.com) reviewed 2024-01-19 08:18:23 -08:00
fuzz/fuzz_layout.cpp
@@ -0,0 +14,4 @@
void FillFuzzedTree(
FuzzedDataProvider& fdp,
const YGConfigRef& config,
const YGNodeRef& root,
NickGerleman (Migrated from github.com) commented 2024-01-19 08:18:22 -08:00
Copy Link

I missed this the first time around, but YGConfigRef and YGNodeRef are themselves just pointers. That also makes const apply to the pointer value, instead of the underlying node/config.

Needed to fix a quick license header to make internal linter happy, but I'm going to make a quick ninja-edit to make that clearer.

I missed this the first time around, but `YGConfigRef` and `YGNodeRef` are themselves just pointers. That also makes `const` apply to the pointer value, instead of the underlying node/config. Needed to fix a quick license header to make internal linter happy, but I'm going to make a quick ninja-edit to make that clearer.
NickGerleman (Migrated from github.com) reviewed 2024-01-19 08:19:03 -08:00
fuzz/fuzz_layout.cpp
@@ -0,0 +14,4 @@
void FillFuzzedTree(
FuzzedDataProvider& fdp,
const YGConfigRef& config,
const YGNodeRef& root,
NickGerleman (Migrated from github.com) commented 2024-01-19 08:19:02 -08:00
Copy Link

Same for a couple other quick repo conventions around naming

Same for a couple other quick repo conventions around naming
nathaniel-brough (Migrated from github.com) reviewed 2024-01-19 09:00:49 -08:00
fuzz/fuzz_layout.cpp
@@ -0,0 +14,4 @@
void FillFuzzedTree(
FuzzedDataProvider& fdp,
const YGConfigRef& config,
const YGNodeRef& root,
nathaniel-brough (Migrated from github.com) commented 2024-01-19 09:00:48 -08:00
Copy Link

No worries. Let me know if there is anything you'd like me to do here :)

No worries. Let me know if there is anything you'd like me to do here :)
facebook-github-bot commented 2024-01-19 09:05:53 -08:00 (Migrated from github.com)
Copy Link

@NickGerleman merged this pull request in facebook/yoga@1a8b80a3d5.

@NickGerleman merged this pull request in facebook/yoga@1a8b80a3d50eacc3251ac1fa2403ce0baff3849e.

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No Reviewers
NickGerleman
Labels
Clear labels
Algorithm
Build and Infrastructure
C#
C++ Enhancement
CLA Signed
Discussion
Documentation
Feature Request
Java
JavaScript
Merged
Question
Reverted
Shared with Meta
Applied via automation to indicate that an Issue or Pull Request has been shared with the team.
YogaKit
YogaLayout ViewGroup
bug
dependencies
Pull requests that update a dependency file
duplicate
enhancement
fb-exported
fh:direct-merge-enabled
invalid
ruby
Pull requests that update Ruby code
wontfix
No Label
CLA Signed
Merged
Shared with Meta
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
DaddyFrosty
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DaddyFrosty/yoga#1537
No description provided.
Delete Branch "add-fuzz-harness"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?