Analysis of Apple Unified Logs: Quarantine Edition [Entry 1] – Converting Log Archive Files on 10.15 (Catalina)

in ,

Please review this post first as it contains an update to this post. Keeping the contents of this blog intact for reference purposes only.

Apple introduce Unified Logging many years ago in 10.12 and has constantly been changing it since its introduction. My main problem is usually using the ‘log’ utility. It has changed over time and those changes are not documented nor is the current documentation adequate in many cases.

My most recent adventure using ‘log’ came when I was running through my course image to create a logarchive bundle from a dead system image. This particular system was running 10.13.1 and will have to be read from whatever macOS the student is running at the time. Silly me for thinking that ‘log’ would always “just work”™️. 

When students upgraded to 10.14.x, they could still create the log archive via the manual process of doing a recursive copy of the /private/var/db/diagnostics and /private/var/db/uuidtext directories into a *.logarchive directory bundle. Works great! Even works (worked?) on iOS devices! On 10.14.x systems we have “force” it into the proper format but it was readable by ‘log’ and the timestamps appeared correct.

When 10.15 came out, this changed. Attempting to create and read the same data produces a corruption error. There is no option to “force” it.

I have been racking my brain and researching this on and off since Catalina was introduced. How the heck can I make this older logarchive readable using a newer versions of macOS?

Comparing the older and newer logarchive formats, they are nearly the same except for a few items. One difference I focused on was the key OSArchiveVersion in the Info.plist. On 10.14 and 10.15 they are on version 4, while on 10.13 they were on version 3. (Note: Doing a recursive copy from a dead image does not create this Info.plist file in the root of the logarchive bundle.)

In my first test, I simply added an Info.plist file to the root of the logarchive bundle with one key, OSArchiveVersion. For the value I tried 4 – why not just give it what it wants. (This testing was done on 10.15.3). 

Great, it parses! However the timestamps are not quite right. I’m lucky to have a dataset that I’m familiar with and have a course labs with specific log entries and their timestamps to validate the conversion. This particular entry I’ve blogged about before – this entry should have the timestamp 2018-02-26 01:49:08.719840+0000. You can see it is only off by a smidge, what gives? These things bother me, so I decided to dive deeper.

Something else is part of this conversion. (I knew it would have been too easy to just change a plist value!)

Next, I changed the value of OSArchiveVersion to a 3. In theory, I hoped to upgrade it to version 4 but still got the same wrong timestamp. I was also not even offered the “force” option. (I really should have known it would have been too easy to just change a plist value!)

In a moment of ‘why not, let me just try this for fun’, I changed the value to 2. …and it actually WORKED! It freaking worked! I had to force it and the timestamps matched up with my original values.

I would love to know why this worked. It makes no logical sense to me.

What happens if I just go straight to version 2 for OSArchiveVersion? It also worked!

A quick command to get this done:

/usr/libexec/PlistBuddy -c "Add :OSArchiveVersion integer 2" galaga.logarchive/Info.plist

What changes in this “forced” conversion? 

I created a new copy of the logarchive bundle and put the Info.plist with OSArchiveVersion = 2. Using one of my favorite file system monitoring tools, fsmon [https://github.com/nowsecure/fsmon]. I was able to monitor what the --force flag changed in the logarchive bundle.

It deletes the timesync directory (and the *.timesync file within) then creates it again with a different timesync file. Also as expected the Info.plist changes from 2 to 4.

What the heck are these timesync files? Not entirely sure, they are binary files that obviously has some sort of time syncing purpose which is why the timestamps now match up. Perhaps I will dig into these another time.

Afterthoughts 

I’m not sure why 10.15 refuses to convert log archives as 10.14 did. Perhaps it is a bug, maybe it is intentional but it sure does make doing forensics difficult. I’m sure this will cause problems with newer versions of macOS as well as other platforms. I’ve heard the same problem exists with iOS, which again will likely be another blog entry when I get to testing iOS specifically.

It does however bring up a good lesson…do not implicitly trust timestamps. I’ll state this over and over until I can get people to test these things. I had it easy with this one because I had known test data – but that doesn’t work so well with random case data.

Below is the mapping of macOS version to Unified Logs Archiver versions (found in the Info.plist files while doing a ‘log collect’ command on a live system.) There are many empty spots – I have no idea if it will help or not but I tried to document it here. If you can help fill it in, I would be grateful please contact me. (Seems most folks update to the latest!)

Log Archive Mapping Table




Introducing 'Analysis of Apple Unified Logs: Quarantine Edition' [Entry 0]

in , ,

I’ve decided to spend some time revisiting analysis of Unified Logs as blog series during this quarantine. It is the perfect topic to make bite sized and I can make it as long or as short as Coronavirus deems it so.

I’m planning of doing smaller blogs at least a couple times a week on a variety of topics such as:

  • Volume Analysis

  • User Logons

  • Airdrop

  • Network Access

  • GUI/Interface Items

  • iCloud

  • Continuity

  • Bluetooth

  • Backups

  • Software Installations

  • Temporal Changes

  • System State & Hardware

  • Siri

  • Apple Pay

  • HomeKit

  • Spotlight

  • Location

Unified logs will give me plenty of things to write about! Looking for something specific, run it by me!

I also plan on creating something like a reference sheet for the queries once I figure out the best place for it.

Let’s get logging!

Analysis of Apple Unified Logs: Quarantine Edition [Entry 1] – Converting Log Archive Files on 10.15 (Catalina)

Analysis of Apple Unified Logs: Quarantine Edition [Entry 2] – sudo make me a sandwich

Analysis of Apple Unified Logs: Quarantine Edition [Entry 3] – Playing in the Sandbox, Enumerating Files and Directories

Analysis of Apple Unified Logs: Quarantine Edition [Entry 4] – It’s Login Week!

Analysis of Apple Unified Logs: Quarantine Edition [Entry 5] – Login Inception!? Yes! – Local Logins!

Analysis of Apple Unified Logs: Quarantine Edition [Entry 6] – Working From Home? Remote Logins

Analysis of Apple Unified Logs: Quarantine Edition [Entry 7] – Exploring USBMSC devices with --style

Analysis of Apple Unified Logs: Quarantine Edition [Entry 8] – Man! What a process!?

Analysis of Apple Unified Logs: Quarantine Edition [Entry 9] – We all know you're binging Netflix! Now Playing on your Apple Devices!

Analysis of Apple Unified Logs: Quarantine Edition [Entry 10] – You down with TCC? Yea, you know me! Tracking App Permissions and the TCC APOLLO Module

Analysis of Apple Unified Logs: Quarantine Edition [Entry 11] – AirDropping Some Knowledge

Analysis of Apple Unified Logs [Entry 12] – Quick & Easy Unified Log Collection from iOS Devices for Testing

Providing Context to iOS App Usage with knowledgeC.db and APOLLO

in , ,

With the APOLLO v1.0 update, I updated many of the Application Activity modules used with the knowledgeC.db database. I mentioned in this article that these were updated to provide more context to specific user application activities. 

One column in particular that was added to all the App Activity modules is Z_DKAPPLICATIONACTIVITYMETADATAKEY__USERACTIVITYREQUIREDSTRING from the ZSTRUCTUREDMETADATA table. I will refer to this as “User Activity Required String” as that is what I’ve named it in the APOLLO modules. 

I will start with some of the application activity parsers for the native iOS apps. The first one, Maps, is particularly difficult but holds some really interesting and useful data.

Maps - knowledge_app_activity_maps

Many of the ‘User Activity Required String’ BLOBs for Maps looks more or less like the following (some are smaller, some larger.) Many investigators will recognize the base64. Whenever you are looking at an Apple device and see base64, you will want to decode it – there is usually great data in there!

You may also notice some readable text at the top of these BLOBs such as in this one. This can help provide quick context, so you can tell if you need to take the time to decode it. This example shows I searched for “Mike Serio’s Po-boys in New Orleans” while at BSidesNoLA this past October. (They were delicious.)

v1.0/com.apple.Maps/t='Mike%20Serio%27s%20Po-boys%20in%20New%20Orleans'&u=

….and others including searches, dropped pins, and directions.

v1.0/com.apple.Maps/t='Show%20DoubleTree%20by%20Hilton%20Hotel%20New%20Orleans'&u=
v1.0/com.apple.Maps/t='Dropped%20Pin%20on%20Nebovidsk%C3%A1%20459%2F3,%20Prague'&u=
v1.0/com.apple.Maps/t='Get%20directions%20to%20The%20Hotel%20at%20Arundel%20Preserve'&u=
v1.0/com.apple.Maps/t='The%20Remedy%20Room%0ADirection%20from%20My%20Location'&u=

I extracted just the base64 string (between the two $ signs) and decoded it on the command line with base64 and then piped it to xxd so I could see what it was. The string ‘BZh11AY&SY’ tells me we are working with a bzip archive file. 

This BLOB has four extra bytes at the beginning that we need to remove before we are able to interpret the bzip archive. We can use ‘tail -c +5’ to easily remove these so we get the bzip header right at the beginning.

Next we will pipe this to bzcat to see what is in this archive, again using xxd last so I can see what it might be. bzcat will extract the contents of a bzip archive to standard out.

Would you look at that, we have a protobuf! I’m not surprised to see this data format; it shows up everywhere! We can parse protobufs with protoc. I will let the reader have fun determining what all is in this protobuf. Might I recommend starting with my article on Apple protobufs.

There we have it, the hoops we have to jump through to get the actual maps data. Thanks Apple, keeping it fun! Here is a quick and dirty recap of my decoding process:

Take the SQLite BLOB ➡️ Extract & Decode base64 ➡️ Remove first 4 bytes using tail ➡️ Unarchive bzip with bzcat ➡️ Decode protobuf with protoc

echo "<base64>” | base64 -D | tail -c +5 | bzcat | protoc --decode_raw

Fortunately, while fun, not all of these apps have crazy base64-bzip-protobuf blobs. Mail, Notes, Safari, Photos, Clock, Weather, and Calendar are relatively easy to visually parse and can provide context to what specifically is happening with the application. Some examples below.

Mail - knowledge_app_activity_mail

Shows access to different email accounts and boxes.

Notes - knowledge_app_activity_notes 

Shows which notes are being edited. This one shows my note detailing what I need for my last vacation.

Safari - knowledge_app_activity_safari

Shows specific Safari browsing data - I’m looking at fancy hotels for my trip!

Photos - knowledge_app_activity_photos

Which photo albums am I viewing, certainly not the one with all my cat pictures. 🤫

Clock - knowledge_app_activity_clock

Did I just cancel my early morning alarm or am I timing how long this meeting is going?

Weather - knowledge_app_activity_weather

Weather may be particularly interesting as it as geo location coordinates embedded. However, they might be someplace I am or am looking to go! You may want to correlate with location data extracted with APOLLO.

Calendar - knowledge_app_activity_calendar

I might be checking meetings or checking how long until the next Objective by the Sea conference is. You should go, it’s in Maui!

maps_requiredstring_calendar.png

3rd Party Apps - knowledge_app_activity

Now we go to the generic App Activity module to review some activities from 3rd party applications. I won’t create modules for every 3rd party application because there are so many. Some applications will have data in knowledgeC.db, some will not.

First up we have my favorite time sink, Twitter. I can see access to specific tweets as well as whichever profile pages you are Twitterstalking.

Dropbox shows access to documents with the Dropbox application.

Venmo shows me paying my debts. The User Activity Required Strings shows interesting information about the contact like their Venmo ID, how many friends they have, friend status, and their Venmo join date. 

And finally, Waze shows me going “Home”.

There is some fantastic and useful information in these activities if you dive into them!