A non-executable stack can prevent some buffer overflow exploitation, however it cannot prevent a return-to-libc attack because in the return-to-libc attack only existing executable code is used. On the other hand, these attacks can only call preexisting functions. Stack-smashing protection can prevent or obstruct exploitation as it may detect the corruption of the stack and possibly flush out the compromised segment.
"ASCII armoring" is a technique that can be used to obstruct this kind of attack. With ASCII armoring, all the system libraries (e.g., libc) addresses contain a NULL byte (0x00). This is commonly done by placing them in the first 0x01010101 bytes of memory (a few pages more than 16 MB, dubbed the "ASCII armor region"), as every address up to (but not including) this value contains at least one NULL byte. This makes it impossible to emplace code containing those addresses using string manipulation functions such as strcpy(). However, this technique does not work if the attacker has a way to overflow NULL bytes into the stack. If the program is too large to fit in the first 16 MB, protection may be incomplete.2 This technique is similar to another attack known as return-to-plt where, instead of returning to libc, the attacker uses the Procedure Linkage Table (PLT) functions loaded in the position-independent code (e.g., system@plt, execve@plt, sprintf@plt, strcpy@plt).3
Address space layout randomization (ASLR) makes this type of attack extremely unlikely to succeed on 64-bit machines as the memory locations of functions are random. For 32-bit systems, however, ASLR provides little benefit since there are only 16 bits available for randomization, and they can be defeated by brute force in a matter of minutes.4
Solar Designer (10 Aug 1997). "Bugtraq: Getting around non-executable stack (and fix)". http://seclists.org/bugtraq/1997/Aug/63 ↩
David A. Wheeler (27 Jan 2004). "Secure programmer: Countering buffer overflows". IBM DeveloperWorks. Archived from the original on 2013-10-18. https://web.archive.org/web/20131018001904/http://www.ibm.com/developerworks/library/l-sp4/index.html ↩
Sickness (13 May 2011). "Linux exploit development part 4 - ASCII armor bypass + return-to-plt" (PDF). https://www.exploit-db.com/docs/english/17286-linux-exploit-development-part-4---ascii-armor-bypass-return-to-plt.pdf ↩
Shacham, H.; Page, M.; Pfaff, B.; Goh, E. J.; Modadugu, N.; Boneh, D. (October 2004). "On the Effectiveness of Address-space Randomization". Proceedings of the 11th ACM Conference on Computer and Communications Security (PDF). pp. 298–307. doi:10.1145/1030083.1030124. ISBN 1-58113-961-6. S2CID 5864467. 1-58113-961-6 ↩