ios

Apple Pattern of Life Lazy Output’er (APOLLO) Updates & 40 New Modules (Location, Chat, Calls, Apple Pay Transactions, Wallet Passes, Safari & Health Workouts)

I started filling in the gaps to missing APOLLO modules. While doing this I realized there was some capability that was missing with the current script that had to be updated. As far as script updates go the following was done:

  • Support for multiple database name -Depending on the iOS version being used the database names may be different but the SQL query itself is the same. Instead of creating many redundant modules I now have it looking for the different database filenames.

  • Support for multiple queries on different iOS versions- You will now notice that all the modules have been updated with iOS version indicators and multiple SQL queries compatible for that version. Some going back as far as iOS 8! I put in as much legacy support as I had data for. I will likely not add much more unless it is by special request. I can’t imagine there is a whole lot of iOS 8 analysis going on out there, but you never know! I have kept it to major iOS release numbers and have tested with the data I have but it may have changed with a minor point release, if you find this to be true please feel free to let me know!

  • Module Timestamp Rearrangement and Module Cleanup– I’ve started to go through some of the modules and move the items around to make it easier to see what is going on with each record. I’ve mostly just moved the timestamps toward the end since most of them are shown in the Key column. I’ve also removed some superfluous columns and extraneous junk in the queries. I’m really only trying to extract the most relevant data. 

A few notes on script usage change. The script flags have changed, with the added arguments of -p = platform (iOS support only for now), and -v = iOS Version. You may also notice the new ‘yolo’ option – this one will likely be error prone if you are not careful. Use this when you what to run it on any database from any platform. It can also be used with your own custom modules if you don’t have versioning in them.

An example of the module changes is below. Notice the multiple databases listed. In this example, the same location data can be extracted from the cache_encryptedB.db or the cache_encrypted A.db databases depending on the iOS versions. The version information is listed in the “VERSIONS” key, while the specific queries have versions listed in the [SQL Query …] brackets, this is the version that the apollo.py script is following.

The big updates were with the modules, lots of new support! I now have support for 129 different pattern-of-life items! Most of the support is for iOS, however if you run the queries themselves on similar macOS databases you will find that many of them will work. Better macOS support is coming, I promise.

Application Specific Usage:

  • Chat – SMS, iMessage, & FaceTime messages extracted from the sms.db database.

    • sms_chat

  • Call History – Extracted from CallHistory.storedata database.

    • call_history

  • Safari Browsing – Extracted from the History.db database.

    • safari_history

  • Apple Pay/Wallet - Extracted from iOS passes23.sqlite database.

    • Apple Pay Transactions - passes23_wallet_transactions

    • Wallet Passes - passes23_wallet_passes

 Location :

  • locationd - The following modules extract location data from the [lock]cache_encryptedA.db & cache_encryptedB.db databases. This will include various cellular and Wi-Fi based location tables as listed in the module filename.

    • locationd_cacheencryptedAB_appharvest

    • locationd_cacheencryptedAB_cdmacelllocation

    • locationd_cacheencryptedAB_celllocation

    • locationd_cacheencryptedAB_celllocationharvest

    • locationd_cacheencryptedAB_celllocationlocal

    • locationd_cacheencryptedAB_cmdacelllocationharvest

    • locationd_cacheencryptedAB_indoorlocationharvest

    • locationd_cacheencryptedAB_locationharvest

    • locationd_cacheencryptedAB_ltecelllocation

    • locationd_cacheencryptedAB_ltecelllocationharvest

    • locationd_cacheencryptedAB_ltecelllocationlocal

    • locationd_cacheencryptedAB_passharvest

    • locationd_cacheencryptedAB_poiharvestlocation

    • locationd_cacheencryptedAB_pressurelocationharvest

    • locationd_cacheencryptedAB_scdmacelllocation

    • locationd_cacheencryptedAB_wifilocation

    • locationd_cacheencryptedAB_wifilocationharvest

    • locationd_cacheencryptedAB_wtwlocationharvest

  • locationd – These modules extract motion data from the cache_encryptedC.db database. Not specific location data but will show device movement.

    • locationd_cacheencryptedC_motionstatehistory

    • locationd_cacheencryptedC_nataliehistory

    • locationd_cacheencryptedC_stepcounthistory

  • routined – Extracts location data from the cache_encryptedB.db database. If you have a keen eye you will notice the database name is the same as from ‘locationd’. Completely different database with different data stored in two different directories.

    • routined_cacheencryptedB_hint

    • routined_cacheencryptedB_location

Health Workouts – Using the healthdb_secure.sqlite database I’ve extracted much of the metadata from workouts. I’ve also determined some of the workout types (ie: HIIT, Rower, Run, Walk, etc), but have not enumerated all of them yet. Please let me know if you come across others – easier if you do this on your own data and can easily look it up. Same for weather conditions (Sunny, Rainy, etc.).

  • health_workout_elevation

  • health_workout_general

  • health_workout_humidity

  • health_workout_indoor

  • health_workout_location_latitude

  • health_workout_location_longitude

  • health_workout_temperature

  • health_workout_timeofday

  • health_workout_timezone

  • health_workout_weather

Network and Application Usage using netusage.sqlite & DataUsage.sqlite iOS Databases

Two iOS databases that I’ve always found interesting (and probably should test more) are netusage.sqlite and DataUsage.sqlite. These two databases contain very similar information – one is available in a backup (and file system dumps) the other only in file system dumps. These databases are excellent at tracking application and process network usage. 

These databases can provide answers to investigative questions such as:

  • What apps were being used?

  • What apps were used more than others?

  • Did the device communicate over cellular or wi-fi more often and when?

  • What apps were used that are no longer on the device?

These databases are located in the following locations depending on the type of acquisition available.

  • /private/var/networkd/netusage.sqlite

    • Available in File System dumps only.

  • Backup: /wireless/Library/Databases/

    • DataUsage.sqlite

    • DataUsage-watch.sqlite (yes, there is one just for the Apple Watch!)

  • File System: /private/var/wireless/Library/Databases/DataUsage.sqlite 

I’ve created modules for these databases in APOLLO, but you can also use the SQL queries in a standalone environment. I’m still working on how best to represent the timestamp keys and may alter the APOLLO code to accept multiple timestamp keys. This will help some other modules I’ve been working on as well so keep an eye out for that. I also need to work on acceptance of multiple database names, thanks to DataUsage-watch.sqlite.

The first set of modules are netusage_zprocess and datausage_zprocess. These two use the same SQL query as it is the same table, just different databases. These query extracts the process and/or the application bundle id. This query will show two timestamps:

  • TIMESTAMP – I believe this is the most recent timestamp for the process/application.

  • PROCESS FIRST TIMESTAMP – This appears to be the first usage of the process/application.

The first example comes from netusage.sqlite, the second from DataUsage.sqlite. It is notable to show that more information is available potentially from DataUsage.sqlite. Since this database is backed up it has the potential to have very historical data. These examples come from my iOS 11.1.2 dump. NetUsage goes back to November 4,2017 when I first setup iOS 11 on the device. The DataUsage database on the same device goes all the way back to 2013! This was from my iPhoneX which certainly did not exist in 2013. I restore backups onto new devices. I also get many more records from the DataUsage database.

netusage_zprocess

datausage_zprocess

The next set of queries are netusage_zliveproces and datausage_zliveprocess. These modules technically have a copy of the ZPROCESS data so they may be redundant if you are running APOLLO. Again, this is the same query for each database. DataUsage will again have many more entries. The added value from the ZPROCESS queries is the added network data information, Wi-Fi In/Out and WWAN In/Out. I will assume this value is stored in bytes until I can test further.

The major difference that I can tell between the two databases (apart from number of records), is that the DataUsage database does not record Wi-Fi network data. I know for sure I was on Wi-Fi at some point in the last six years! (It shows in NetUsage – remember it is the same device. Check out my Twitter data, its almost horrifying! 🤭)

netusage_zliveprocess

datausage_zliveprocess

Finally, we have an additional query only for the netusage.sqlite database, netusage_zliverouteperf. This query extracts lots of information, some of which I have no idea what it is. The first step into determining this is creating the query! In addition to some timestamps that appear to be stored on a per-hour basis we have the type of network traffic (Cellular or Wi-Fi), bytes and packings coming and going, connection information.

A second screenshot is required to show the rest of the extracted data. Some sort of cellular network identifier (any ideas?) or Wi-Fi (SSID/BSSID) are provided, with additional network-based information.

There is a lot of data going through these pipes!

On the Twelfth Day of APOLLO, My True Love Gave to Me – A To Do List – Twelve Planned Improvements to APOLLO

My Christmas gift to you - improvements!

  1. More Queries – There is plenty more to come. There are more databases and many half-written queries that I have yet to add.

  2. Additional Testing – I want these to be as accurate as possible.

  3. BLOB/Protobuf Parsing – More location information is useful.

  4. Plist Extracting – So much metadata that puts more context to the data.

  5. Database Coalescing – Those WAL files are important.

  6.  Data Visualization – Pretty pictures always help.

  7. Unarchiving of Powerlog Archive Files – Can’t forget about those archives!

  8. More macOS Coverage – I’ve been focusing on iOS, but there is some good macOS databases too.

  9.  Version Detection for Different SQL Queries

  10. Potentially Different Output Formats – Any special requests?

  11.  Better Module Documentation – Describe what each query is extracting in the module notes.

  12. Better Activity Categorization – More specific categories for better filtering.

Grab APOLLO Here!

Start with Day 1: On the First Day of APOLLO, My True Love Gave to Me - A Python Script – An Introduction to the Apple Pattern of Life Lazy Output’er (APOLLO) Blog Series

On the Tenth Day of APOLLO, My True Love Gave to Me – An Oddly Detailed Map of My Recent Travels – iOS Location Analysis

I saved one of my favorite topics for (nearly) last. There is no question that location can play a major role in many investigations. 

iOS location data as changed drastically with iOS 11 from previous iOS versions. I published research on these locations in the past and parsing scripts.

It is my goal to update these scripts with this new research soon(ish).

Powerlog Metadata

The CurrentPowerlog.PLSQL contains some useful metadata associated with the primary locations data I’ll discuss a bit later in this article.

The powerlog_location_tech_status module contains a log of how location was determined. Was the location determined by Wi-Fi or GPS technologies? This information can contribute to how accurate the location data may be.

The powerlog_location_client_status module appears to keep track of which applications and services are requesting location data. Some app examples below include Waze, Weather Underground, The Weather Channel, and the AUDI app. The services can be seen in the second screenshot (the data contained in this table was too long for only one!). 

The type of location is also recorded along with accuracy figures. I’ve seen the following types.

  • Location

  •  Significant (Likely has something to do Significant Locations)

  • Fence (Geofencing?)

  •  Visit

Finally, we have a small log of providing time zone context to the data. The powerlog_timezone module will extract this information.

Significant Locations & Routined Databases

As I’ve mentioned above, the storage of the routined process locations and Significant Locations has changed dramatically in iOS11 from previous versions.

These databases are stored in the following path and are only accessible in full file system dumps.

  •  /private/var/mobile/Library/Caches/com.apple.routined/

    • Cache.sqlite

    • Cloud.sqlite

    • Local.sqlite

Cache.sqlite - routined Locations 

The first database, Cache.sqlite, contains an extremely detailed history of coordinates where the device was. In my own data I had over 40,000 (!) data points. This data is stored for just over a week. This can provide a very accurate map of where I was during that last week.

The routined_cache_zrtcllocationmo module can extract these coordinates along with a timestamp, altitude, course, speed (meters/second), and vertical/horizontal accuracy figures.

Also kept for about a week is data extracted with the routined_cache_zrthintmo module. It has fewer data points, but the timestamp and coordinates appear to be accurate.

Cloud.sqlite - Significant Locations - Visits

The next few modules all use basically the same query but are keyed off of different timestamps. The example shown is of the first module.

Again, this screenshot was split into two because of the amount of data provided by this table. Each significant location visit contains various timestamps (visit entry/exit, visit creation/expiration, learned place creation/expiration). Each visit has coordinates along with a Place ID (an identifier for a specific location), data points collected for that place, uncertainty and confidence figures, and device logging information. 

In the second screenshot there are two odd looking columns, Place Name BLOB and Place Geo BLOB. Each of these columns is storing BLOB data in hex format. I chose this format as it is easy (relatively) to copy/paste into a hex editor for viewing. Examples below.

The first column “Place Name BLOB” contains a smaller amount of binary data (than the Place Geo BLOB), and as you can see you can fairly easily determine where I was at this time – Dulles Airport (I’m a frequent visitor there as you can imagine.) You get address, city, state, business name information in this blob.

The second column “Place Geo BLOB” contains much more binary data, but you can still pick out some similarities in the strings.

So here is the kicker – for YEARS I just accepted these BLOBS, tried to parse them and was unsuccessful but didn’t care too much as I could always see the contents of them. While putting this article together I finally discovered their format – it’s a Christmas miracle!

I do lots of mobile work, that includes Android devices for a good portion of the time (Android, eww – I know, but it provides a paycheck. 😉) When I see random BLOBs in anything Android related my first inclination is to say it’s a Protobuf – a very Googly format. 90% of the time I’m right, I can usually spot them pretty easily. For some reason while looking at them now, my usual protobuf triggers just slapped me in the face so I gave the protoc a try (see the usage below.) 

protoc --decode_raw < binary_BLOB

It worked! I finally have a parsed data structure, granted some of the protobuf pieces I still need to determine but its certainly further than I’ve gotten before. Below is the protobuf output for the “Place Name BLOB”. Many of the strings are obvious, however the highlighted section is the coordinates for Dulles Airport. These are 8-byte floats stored in big-endian as shown in the hex editor below.

The “Place Geo BLOB” is quite a bit lengthier, too long in fact to create a decent screenshot of but still contains the string data and coordinates as expected.

Fun Fact – These protobuf data BLOBs are scattered throughout macOS and iOS systems, you can also see them in Maps data if you are looking for examples to play with. Those also appear to contain timestamps! I promise I’ll do a nice protobuf location blog in the near future. I love me some protobufs! (I’m weird, I know.)

Local.sqlite - Significant Locations - Locations of Interest

Ok back to Significant Location databases. There is one left, the Local.sqlite database. This contains similar data to what we’ve seen before. Timestamps, coordinates, confidence, uncertainty, data point count, and (protobuf!) BLOBS. As with the other modules, they query is basically the same but is using different timestamps to key off from. The example is from the first module. These screenshots are also from the same query, too long to post as a single screenshot.

The last piece of location data I’m extracting from the Local.sqlite database is parking history. I connect my iPhone to my vehicle via CarPlay so I’m not sure if this is required to populate this data. The first module routined_local_vehicle_parked shows the last location of my parked car. 

The routined_local_vehicle_parked_history module shows a parking history.

These modules should give you a pretty good idea where a certain user’s device was a given moment. It is worth mentioning the data does expire after a certain period of time, the sooner you have the data the better the data will be for you. Historically it appears to be pretty accurate but significant locations, but it doesn’t have every location. Dump that phone ASAP! 

Santa can easily find where you live. Creepy Santa.

Grab APOLLO Here!

Start with Day 1: On the First Day of APOLLO, My True Love Gave to Me - A Python Script – An Introduction to the Apple Pattern of Life Lazy Output’er (APOLLO) Blog Series