Project ideas

1. Sometimes a vulnerability might be patched without releasing any details about how it can be exploited. Can you find such a vulnerability and exploit it given the patched and unpatched version of the program?
2. Fuzzing is an automated software testing technique that involves providing invalid, unexpected, or random data as inputs to a program. An effective fuzzer generates data that is valid enough so as to not be directly rejected by the program but invalid enough so as to create unexpected situations that are not properly dealt with in the program. Write your own small fuzzer that is able to generate and provide input to other programs and log whenever they crash.
3. Write an parser for executables that outputs relevant information about the executable similar to 'readelf' (ELF) or 'dumpbin' (PE – Windows executables). You can further extend this project by writing a loader for executables. It doesn't have to be a very complicated loader, being able to run executables that have no dependencies is more than enough.
4. Some executables have their code encrypted and only decrypt it at runtime. In order to inspect the real code, one needs to reverse engineer the decryption routine and statically decrypt the code. Another way would be to let the program decrypt itself. This can be done by debugging and breaking after everything is decrypted, but some malicious programs use a lot of anti-debugging tricks that make this a real pain. Another way of doing it is letting the program run and dumping its memory after a small amount of time. Your task is to write such a tool that dumps the memory contents of another process. You will probably need to hook the exit() function if the process ends too quickly and your tool doesn't get the chance to dump its memory. This can be done for Windows as well.
5. Write a basic binary debugger to debug other applications with. You might need a disassembling library for your debugger, you can use Capstone (http://www.capstone-engine.org/). This can be done for Windows as well.
6. A control flow graph (CFG) is a representation using graph notation of all paths that might be traversed through a program during its execution. Sometimes it's desirable to be able to look at the CFG of a program. Try and build a tool that generates the CFG of a binary. One disassembly library you can use for this task is Capstone (http://www.capstone-engine.org/).
7. Use a fuzzer (e.g. AFL to find crashes in an open-source project. Evaluate which might be security vulnerabilities.
8. Use a static analyzer (e.g. clang static analyzer, Facebook Infer) and try to confirm some of the warnings by finding actual inputs.
9. Use dynamic taint tracking and fuzzing (e.g., Vuzzer: code, paper) on programs with (formatted) file input. Log what file data (offsets) have been accessed and find causes of crashes (missing checks).
10. Use binary instrumentation with Intel Pin to track program behavior, for the same purpose as above.