[QUESTION]How to use leak detector client?

[renzhengeek]

Hi all,
After building leak detector successfully, I try to use leak_detector.

1. everything seems okay util initialize granay, like the following:

[ 1122.265119] [granary] Relay channel initialised.
[ 1122.265122] [granary] Done; waiting for command to initialise Granary.
[ 1122.265125] [granary] Notified of module 0xffffffffa07b98a0 [.text = ffffffffa0195000]
[ 1122.265126] [granary] Module's name is: granary.
[ 1122.265128] [granary] Ignoring module state change.
[ 1570.599216] [granary] Initialising Granary...
[ 1570.623929] [granary] Initialised.
[ 1570.641799] leak_policy_update_rootset
[ 1570.656410] leak_policy_scan_callback, ffffffffa25f1300
[ 1570.656412] No allocation in the last epoch

1. when insert the following target module, target VM reboot itself immediately.
2. how to use and debug leak detector?

All the best for you.

[pgoodman]

I assigned Akshay to this because I have no idea how it is used.

Try invoking sudo touch /dev/granary after initializing Granary. After Granary is initialized, every invocation of sudo touch /dev/granary leads to client::report being invoked. That being said, I am not sure if Akshay tied things in to client::report.

[renzhengeek]

Okay,thanks:)

[kumarak]

The granary initialization log is not showing much about the issue. Can you add printf statements inside leak_policy_scan_callback, and see how far it is making there?

I can see two reason for reboot. One the instrumentation policy for detecting when to add watchpoint is not correct or the leak scanner is accessing some invalid data. Can you try to verify each of them separately? Disable leak scanner thread by commenting part of code from init. If the issue exist the problem is with instrumentation. Otherwise the scanner is surely accessing some invalid data. May be descriptors is not initialized.

[kumarak]

If there is any changes from your side, discuss with peter and checkin the changes in some temporary branch. I can look into that.

[pgoodman]

First, recompile with CONFIG_DEBUG_CPU_RESET set to 1 (https://github.com/Granary/granary/blob/master/granary/globals.h#L100), then attach gdb before Granary's initialization and check to see if gdb can catch the CPU reset when you load your module.

If this does not work, then make sure that your qemu.sh specifies -d int,cpu_reset. This is a debugging flag that tells QEMU to log CPU resets. Before loading your module, copy down the address at which Granary is loaded. This address is found in the system log. For example, in the below log, Granary is loaded at address 0xffffffffa0195000.

...
[ 1122.265125] [granary] Notified of module 0xffffffffa07b98a0 [.text = ffffffffa0195000]
[ 1122.265126] [granary] Module's name is: granary.
...

Then, load your module. If the VM reboots then open up /tmp/qemu.log (on your host machine), and copy the contents of that file here. To keep it brief, omit blocks of that log where all dumped registers are 0.

What we're looking for in the /tmp/qemu.log is the value of RIP (instruction pointer) at the instruction that led to the CPU reset. We can't always do much with this information. For example, if the address is a code cache address, then we're likely out of luck, unless it's "gencode" (e.g. patch wrappers, clean callable-generated code). Anyway, subtract Granary's loaded address (from the system log) from the value of RIP (from QEMU's log). Hopefully this value will represent an offset into the Granary binary. Then, run objdump -d bin/granary.ko > /tmp/granary.S. Convert the offset into hexadecimal, open up /tmp/granary.S, and see if you can find the code associated with that offset!

[pgoodman]

Setting the CONFIG_DEBUG_CHECK_CPU_ACCESS_SAFE macro to 1 would also be prudent (https://github.com/Granary/granary/blob/master/granary/globals.h#L95).

[renzhengeek]

Hi kumarak,
Now changes in leak_detector code can be seen by
git diff leak_detector:clients/watchpoints/clients/leak_detector/ origin/release:clients/watchpoints/clients/leak_detector/

leak_detector branch is now in my repo:https://github.com/renzhengeek/granary/tree/leak_detector.

Thanks.

[pgoodman]

Let me know if the debugging suggestions above reveal anything new.

[renzhengeek]

Okay,but it will take some time :)

[renzhengeek]

Hi pgoodman,
It cannot hit breakpoint,and reboot.here is the content of qemu.log:

CPU Reset (CPU 0)
EAX=fffffffe EBX=00000000 ECX=00000000 EDX=00000000
ESI=0000000a EDI=fffffffe EBP=00000000 ESP=00000fc4
EIP=000f4f6a EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
SS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
DS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
FS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
GS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
TR =0000 00000000 0000ffff 00008b00 DPL=0 TSS32-busy
GDT=     000fd3a8 00000037
IDT=     000fd3e6 00000000
CR0=60000011 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
CCS=00000000 CCD=00000000 CCO=DYNAMIC
EFER=0000000000000000
FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80
CPU Reset (CPU 1)
EAX=00000000 EBX=00000000 ECX=00000000 EDX=000206a7
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 0000ffff 00009300
CS =f000 ffff0000 0000ffff 00009b00
SS =0000 00000000 0000ffff 00009300
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT=     00000000 0000ffff
IDT=     00000000 0000ffff
CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
CCS=00000000 CCD=00000000 CCO=DYNAMIC
EFER=0000000000000000
FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80
CPU Reset (CPU 1)
EAX=00000000 EBX=00000000 ECX=00000000 EDX=000206a7
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 0000ffff 00009300
CS =f000 ffff0000 0000ffff 00009b00
SS =0000 00000000 0000ffff 00009300
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT=     00000000 0000ffff
IDT=     00000000 0000ffff
CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
CCS=00000000 CCD=00000000 CCO=DYNAMIC
EFER=0000000000000000
FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80

[pgoodman]

This seems unusual. The output isn't showing all 16 general purpose 64-bit registers; it appears as if it's showing the 32-bit machine state.

What about this: comment most of the code out until you hit some critical part where it un/commenting some feature breaks things.

For example, start by commenting out most of the kernel-specific code in kernel/leakpolicy_scan.c. Then the wrappers. Then comment out the code in the leak enter/exit functions that are injected into the code.

[renzhengeek]

Hi kumarak,

1. I set breakpoint at client::wp::leak_policy_scan_callback, it don't hit this breakpoint before reboot;
2. you say commenting part of code from init, Can you specify the location of the code;
3. here is the temporary branch: https://github.com/renzhengeek/granary/tree/leak_detector, and you can see changes by git diff leak_detector:clients/watchpoints/clients/leak_detector/ origin/release:clients/watchpoints/clients/leak_detector/;
   4.can you talk about leak_detector client further? for example,TODO list about it.
   thank you:)

[kumarak]

if your breakpoint is not getting hit then removing scanner thread will not be of much help. But still you can try commenting following line from instrument.cc.
leak_policy_scanner_init(scan_callback, update_rootset_callback);

I suspect the issue is because of illegal memory access, I will download your code base today and look into that.

[pgoodman]

Please fix https://github.com/Granary/granary/pull/27/files#r11557171