Source: http://blog.spiderlabs.com/2014/10/jailbreak-detection-methods.html
Many iOS applications contain some sort of jailbreak detection mechanism. Some of the detection mechanisms can be bypassed by attackers (sometimes easily), whereas others are quite difficult to bypass. Below is a list of some of the more popular methods of detecting jailbroken iOS devices.
1. Filesystem-based Detection
The jailbreak process modifies the filesystem by adding, moving and changing files and directories. These changes can be detected, to determine if the device is jailbroken or not.
New Files Created
During the jailbreaking process, some additional files are created on the device. Looking for these files is a simple way to detect a jailbreak. It's also an easy method for a malicious individual to detect and bypass. An attacker can search for a string in the application, and then simply change the file names in question to avoid detection.
The most popular files that jailbreak detection is based on are listed below:
/private/var/stash
/private/var/lib/apt
/private/var/tmp/cydia.log
/private/var/lib/cydia
/private/var/mobile/Library/SBSettings/Themes
/Library/MobileSubstrate/MobileSubstrate.dylib
/Library/MobileSubstrate/DynamicLibraries/Veency.plist
/Library/MobileSubstrate/DynamicLibraries/LiveClock.plist
/System/Library/LaunchDaemons/com.ikey.bbot.plist
/System/Library/LaunchDaemons/com.saurik.Cydia.Startup.plist
/var/cache/apt
/var/lib/apt
/var/lib/cydia
/var/log/syslog
/var/tmp/cydia.log
/bin/bash
/bin/sh
/usr/sbin/sshd
/usr/libexec/ssh-keysign
/usr/sbin/sshd
/usr/bin/sshd
/usr/libexec/sftp-server
/etc/ssh/sshd_config
/etc/apt
/Applications/Cydia.app
/Applications/RockApp.app
/Applications/Icy.app
/Applications/WinterBoard.app
/Applications/SBSettings.app
/Applications/MxTube.app
/Applications/IntelliScreen.app
/Applications/FakeCarrier.app
/Applications/blackra1n.app
Directory permissions
Like detecting a jailbroken device by looking for certain new files, certain permissions on partitions and folders can also indicate a jailbroken device.
During the jailbreaking process, access to the root partition is amended. If the root partition has read/write permissions, the device has been jailbroken.
Size of /etc/fstab file
The /etc/fstab file contains mount points for the system. Many jailbreaking tools modify this file by adding entries to it, changing its file size. The typical iOS app isn't capable of reading the file, but it can check the size of the file.
Do note however, that the file size can change as a result of a new update from Apple.
Existence of symbolic links
Some directories are originally located in the small system partition, however, this partition is overwritten during the jailbreak process. Therefore the data must be relocated to the larger data partition. Because the old file location must remain valid, symbolic links are created. The following list contains files/directories which would be symbolic links on a jailbroken device. An application could check for these symbolic links, and, if they exist, detect a jailbreak.
/Library/Ringtones
/Library/Wallpaper
/usr/arm-apple-darwin9
/usr/include
/usr/libexec
/usr/share
/Applications
Writing files
On jailbroken devices, applications are installed the /Applications folder and thereby given root privileges. A jailbroken device could be detected by having the app check whether it can modify files outside of its sandbox. This can be done by having the app attempt to create a file in, for example, the /private directory. If the file is successfully created, the device has been jailbroken.
2. API-based Detection
Some API calls provided by iOS behave differently if run on jailbroken devices. Detecting a jailbroken device based on API calls can be both effective and difficult for a malicious individual to recognize and bypass.
fork()
The sandbox denies process forking on non-jailbroken devices. By checking the returned pid on fork(), an app can detect if it has successfully forked. If the fork is successful, the app can deduce that it is running on a jailbroken device.
system()
Calling the system() function with a NULL argument on a non-jailbroken device will return 0. Doing the same on a jailbroken device will return 1. This is because the function will check whether /bin/sh exists, and it only exists on jailbroken devices.
vm_protect() - OBSOLETE
On iOS versions prior to 4.3.4, memory pages could not be marked as executable if the device was not jailbroken. Later versions of iOS have changed this rendering this detection method obsolete.
dyld functions
This detection method starts with calling functions like _dyld_image_count() and _dyld_get_image_name() to see what dylibs are currently loaded. This method is very difficult to dynamically patch due to the fact that the patches themselves are part of dylibs.
Attackers have a difficult time bypassing this detection method.
3. OpenSSH Service Detection
Jailbroken devices can run services that aren't normally present on non-jailbroken devices - the most common is the OpenSSH service.
Note that this detection method can be very slow. If SSH is not installed or running on the device, it can take some time for the connection to timeout. Attackers can also easily bypass this method by simply changing the port for the OpenSSH service.
4. Cydia Scheme Detection
Most jailbroken devices have Cydia installed. While an attacker can change the location of the Cydia app, it's unlikely they will also change the URL scheme the Cydia app is registered with.
If calling the Cydia’s URL scheme (cydia://) from your application is successful, you can be sure that the device is jailbroken.
It's difficult to change the scheme for Cydia, but it is possible to simply remove Cydia during the testing process.
5. Summary
In general, the more complicated the jailbreak detection is, the more difficult it is to detect and bypass. The most common mistake when implementing jailbreak detection often lies in the implementation itself.
We often come across apps that have great jailbreak detection, but the implementation is in one function that returns true or false depending on whether the device is jailbroken. In these cases, we bypass jailbreak detection by using Cycript or a similar tool to invert the return value from the detection function.
In practice, the best jailbreak detection combining multiple techniques and integrating them into other functions so that they cannot easily be bypassed.