[Music]
hello everyone welcome back to the very
last talk for this year's linux
conference
all right to wrap things up on a
fascinating note um we have indu pagat
nick alcock telling us about compact c
type format in the
uh neutral
tool chain uh tool chain was like the
easiest word to pronounce in that entire
title and i messed it up okay
uh indo is part of the linux tool chain
group at oracle in the recent few years
her focus has been on ctf btf support in
the neutral chain uh nick who is not
able to
join us sadly due to time zones
is a free software developer currently
working on dtrace ctf new bin utils and
other tool cheney things at oracle
so yes nick is not able to join us but
indo is here
this is a pre-recorded talk so indy will
be joining you all in the chat
um
so feel free to
have a nice time chatting away during
the talk um in duty would you like to
say a few words
uh i welcome everyone and i'm grateful
to be here presenting this
exciting talk about compact c type debug
format um i hope you guys enjoy the talk
hello everyone welcome to the talk on
cdf the compact c type format and gnu
tool chain i and nick will be doing the
talk jointly today
so let's get started for those of you
who have not heard of ctf before
hopefully this is an exciting and useful
talk for you we will talk about
what cdf is all about and how to
generate and use it if you have heard of
and played with ctf before
i hope this talk will still be exciting
for you
because we will talk about adding useful
new extensions to the format which will
be included in ctfv4
so let's get started by first looking at
what is debug information
debug information is the metadata that
conveys relationship between the
executable version of your program and
the original
source code of your program it includes
information around
types of program construct source
location information
information for virtual stack unwinding
and so on so forth
there have existed many debug formats
and many of you would have heard of or
even used dwarf it is the most commonly
used
debug format it is very effective
for elf executables
dwarf
debug information is available
via
the debug underscore sections
dwarf is designed for expressiveness and
is very powerful it works for a variety
of platforms apis and languages it
encodes
types source location information on
wine
information call site information and
much more it's quite comprehensive
but there are some issues with dwarf
it is large and in practice it is
voluminous enough that most of the time
it is stripped out or shipped as a
distinct package or not shipped at all
uh it is complex interpreting some
aspects of dwarf need you to implement a
small stack machine or have access to a
small stack machine where
these dwarf op codes need to be executed
and dwarf expressions need to be
evaluated
and it's not just the
the
it's not just the dwarf expressions or
the need to have a
stack machine that adds to the
complications there are other aspects of
dwarf that sometimes make it too bulky a
format to get the job done it is slow
and as we will talk uh later today
for a specific use case of online
debugging uh dwarf based methods are not
the first choice
so here comes cdf with its niche
offerings
cdf is compact and fast ctf can
represent types
for all c constructs it can be used to
provide a c application
full runtime visibility into its type
system the second most important feature
which is currently work in progress and
we will talk about this in the latter
half of the talk is generating back
traces
cdf um aims to provide a fast compact
way to generate back traces
the support is being targeted to be
present in the tool chain so putting all
this together ctf can help alleviate
some of those uh issues
uh with dwarf by providing
some means of debugging if dwarf methods
are not preferred or not possible
so
what is ctf cdf stands for compact c
type format and as the name goes it
describes c types of a program it
originated in the solaris kernel in the
mid 2000s to describe the kernel debug
information and was later ported to
linux
what we have today is the linux ports
version 3
so ctf v3
dtrace on linux has been a consumer of
ctf
but but cdf in general is a debug format
it is independent of d trace
the ctf spec is available at this
location
and also in
bit neutrals master
now because ctf and dwarf are debug
formats sometimes this question of how
do the two for debug formats compare
does come up and to that question my
opinion is that this is not a meaningful
comparison
ctf and dwarf serve
different set of use cases dwarf is the
most comprehensive debug format that
serves the need of most offline
debuggers
cdf on the other hand is type
information for c only at this time
it does not have any location
information no complex expressions to be
evaluated hence no stack machine in the
reader etc
so dwarf encodes a lot more information
and the format itself is very different
from ctf even for presenting just types
and ctf on the other hand is a more
compact simpler format
which makes it faster to load so
overall i would still say that this is
not a meaningful comparison um but so
let's see in the next few slides i will
describe um the cdf formats so if you
know dwarf you'll be able to appreciate
the differences between the two formats
and hopefully you'll have an opinion of
your own on this matter
so let's get started with
some details on ctf debug format um
apart from the spec ctf format is also
documented in ctf.h header file
installed by bennutils
at the heart of cdf section is a ctf
dictionary which is just um which is
just a name for a set of cdf types
these dictionaries can be arranged in a
parent-child relationship called ctf
archives
for much of the discussion today we will
focus on ctf
as it is the crux of the cdf section
so each cdf dictionary has multiple
subsections where each subsection stores
type information for a class of c
constructs
so if you if you can look up the if you
if you want to look up the type of the
variable in your program
you can follow the
variable subsection similarly you can
know the return type and the argument
type of um a function in your program by
following the
function subsection
all the strings for variable names
struct names etc they appear in the ctf
string table which sits at the very end
of the ctf dictionary
the cdf types subsection is an array
of variable
length entries so
each of these struct cdfs type is
followed by variable optional variable
length data and each type has an id
which is simply its index in the array
so type isd is not explicit in the ctf
format
ctfs type underscore t struct is used to
encode the basic information for any
type in cdf so let's see what it looks
like
ctt name here stores an offset to
the name in the ctf string table
each type needs to have either a size by
itself or refer to another type so for
example if it's a typedef it will refer
to another type via the ctt type
but it will not just have a size right
here ctd info
let's see cdv info is in effect a tiny
struct itself which encodes the kind of
the type
uh so it tells you whether it's an array
pointer type def struct integer
and all that and the number of
subsequent entries to expect after ctfs
type struct right so there needs to be a
way where each type
where the reader of ctf type knows what
follows cdfs type underscore t and ctt
vlan is just that
so
the number of entries that follow a
specific type depends on the kind of
type some in some types like integers
have a fixed number of bytes that follow
other types like structs and functions
have a variable number of struct members
or you know function arguments
respectively
each of which gets an error element
giving its name type and offset so
maybe let's see some examples to to put
it uh perhaps more clearly so let's
with that background let's put it
together for integer data type
as you see ctd name points to the string
int in ctf string table ctd info tells
you that the kind of type is ctfk
integer ctt vlan is set to 0
and a cdf s type underscore t the first
block is immediately followed by a
variable length data which for end is a
32-bit
data data stream it encodes in this case
whether the
integer is assigned one a
signed integer or not and other details
okay let's work out another example for
array
representation in cdf and array type in
cdf is represented by using a ctfs type
underscore t
followed by a cdf underscore array
underscore t object so in this case the
cdt name points to a null string in the
ctf string table cdt info tells you
this
type is of kind ctfk array
ctd vlan is again set to zero and unused
the variable length data in this case is
a struct cdf array underscore t which
specifies a reference to the type of the
array
and the number of elements so
that's it putting it together the cdf
types refer so this pictorially shows
you that for a sample uh c code this is
the kind of types you will see in your
ctf type subsection the
and
so the tcdf types reference
so in this diagram here the ctf types
reference each other to create a
representation of the types in your high
level program as shown
and in this pictorial layout here on the
right side the types in white are
basically base types the pink ones are
pointers
and the yellow ones are sort of
composite type structures and so on so
all this helps give you an idea of how
these types weave together to give you a
true representation of all the types in
your program
so so far we've only been dealing with
single object files but then no one runs
a single object file executable
and shared libraries what people are
interested in
so
we did with this by having the link of
spot ctf in in in incoming object files
and duplicate duplicate it all together
um so that so that no type appears more
than once in the output um
the layer also hunts around and finds
symbols that okay and
and find all the all the symbols that
will appear on the output and if they
have types and that have known types um
it builds a table so that ctf users can
associate the types without the um can
ask given this symbol what is it what is
its type i'm going to type back
um this is all in the light implemented
in the library the link it uses it
anyone else can use this as a can use it
as well it's not tied to generally in
any way anyway except that it appears in
video tools
the duplication is fast
most of the timing
on the next slide it seems to be
reliable it saves a lot of space
um
but there's a wrinkle
c is annoying unlike c plus plus there
is no one definition rule types can have
different it can have the same name but
completely different definitions in
different translation units we spot
these types and any types
any types they relate to and put and put
them into in separate child dictionaries
um
which have uh which which are connected
which are connected to the uh to the
parent and [ __ ] and and and and if you
look at if you look for a type in the in
the child you can see all the parents
types as well
um if child if ambiguous types are
present we emit an archive of ctf
dictionary dictionaries into the um
output executable or shared library
rather than a single dictionary but this
is mostly transparent to the user
um as in even if there is no even if
there is no archive present the api
makes it appear to be a single single
member archive
um
we because we merge all the cts into a
into a single section we do not track
which translation units define which
types debugging users mostly don't care
and it would be an enormous
expensive space because often most
translation units define most types
here's an example
of ambiguous types in this situation
structure the foo is a structure in one
translation unit and and to give it 16t
in the other
um and and you can see and you can see
here that all the types end up in the
shared.ctf
archive member except for except for the
one which is varying which appear um
which appears in in both um
in the appropriate perfect um person you
child you know that the co has
disappeared entirely it hasn't um it has
nothing which isn't present
in other um it has nothing ambiguous and
so it just all it types diffused into
dot ctf and it vanishes ld can be asked
to distribute types in other ways but
this is uses far less space than the
other approaches that people are likely
to want to use almost all the time
the result is pretty good as soon as as
far as poses go all the phrases here are
faces of the dot pts sections and
they're functions of uncompressed dot
ctf sections ctf is almost always
compressed unless it's tiny currently
with jesus um i'm only doing
uncompressed places to make it uh to
show you just how large the reduction is
um so the ld drops with 3.2 meg of ctf
on the uh on the input to 200k on the
output and 60 70k after compression emex
drops from seven meg um to 220 all the
way down to 129k
and in many cases the at the output if
you look at lymphedema the output is
smaller than the uh the
the
single object files worth of the input i
think that's fairly acceptable
compression
um time-wise it takes almost no time at
all even though it's single-printed and
not really optimized there's a lot of
low-hanging fruit to improve there as
well
um
what's the support like in the tool
chain it's on gcc 12 master plus n minus
gcts
um
it's in canoe ld you don't need to pass
anything if the tt is on the on the
input you get you get compressed uh and
um you duplicated ctf on the app but
it's in object dump it's in readout in a
moment
um it's not in gold yet i'll have i'll
have to add that at some point um and
uh and
and qld does not um support non-elf
platforms with ctf at the moment it just
hasn't been plumbed in
um there's no support in llvm at all it
just hasn't been added people have been
asking occasionally
it might get it should get done
eventually it might get done eventually
people have also been asking for c plus
that might get done eventually as well
gb we support uh we support cta um we
support cdf dictionaries fully p type
works obviously question obviously tv
features don't work without dwarf but if
you want insulin types we can give it to
you
um there are other things gdp users
would be likely to want and that's what
the main subject of this talk is about
um
there's more support glue poke can rea
can read ctf um
it's actually a
very nice at this sort of thing um i use
it in development these days um the
abigail
can take in executables that contain
only cdf
um
and do api analysis of them and if we
end up with most executables containing
ctf which we might visit small this
means you'll be able to see whether
things are api compatible without
installing any kind of debug info at all
it also means that things might be able
to might be able to check for api
compatibility at runtime they just need
to call on call and go to ctf and ask it
or live
so a quick demo
turn on
we have here two really exciting
translation units um one to um
they contain almost the same almost the
same content except that except accepted
in one case um
struck foo has an extra member
um structural too annoying because it
points to itself um
cycles are a known bane of
systems like this
what happens what happens if you link
this together let's try it
here we have some cts that's too bad
we can't dump it
just want to see what it looks like
i think maybe i should make this
window full size
and
this this is the the dot the cdf section
the the the
the main dictionary there's all sort
there's all sorts of stuff in here
most significantly there is an i o there
is an i o file we use studio so there's
a file star in there
completely represented
um
all the pointers and so on i shared the
only unshared things in the individual
archive members down here
are the actual structure which differed
here
here it is it won't see you with fubar
and next
it is and the other one with
zubar there's two bar badly the the new
extra integer and next it's pretty
duplicated about as well about as well
as you could hope and all that all the
types have to be represented if you look
here
data object it knows that still there is
a file star and it knows what sort of
thing a file star is and it knows it
turns into a struct i o file
um despite the fact that we didn't
bother
to tell any any of that gtc and we did
it for us
so this is all it's still done with an
underlying api um i think other people
could use it it's divided into several
sections you can query the properties of
types you can open and click you can
open you can open else objects and you
get a ctf dictionary back or a cdf
archive you can dig over the archives
and find types in them and that sort of
thing you can walk over struck members
types anything anything in ctf you can
integrate over it you can create you can
you can add members you can create them
the ctf knows everything about how to
create ctf dictionaries so um
so you never need you should never need
to write your own code to do it or you
can if you'd like to there is a
um and you can link them together in
about in about five lines of code which
is which is quite nicely mean you can
write plenty linkers in about in about
200 lines um and there's plenty it would
be about 10 lines if it hadn't been it
wanted to do exactly what google d did
but in that case you can just use google
ld
uh what's coming
ctsd4 is being planned for uh the big
thing of course backtrack support um
which is halfway between ctf and the
user of cts really because it has to
work with note ct we have no need for
the caller to read ctf at all
um that's what the rest of this talk is
about but also we're planning to improve
expressiveness because there are some
things in the pipe system that we're
going to see type system that we can't
represent yet all
syntax extensions and that sort of thing
um we can improve compactors in various
ways reduce the space used to the string
table by cutting it into chunks um use a
better compressor than cheese if we're
thinking since we're thinking zedsted um
there's a bunch of things planned we
even have some plans for each side
because we don't want to do everything
all at once in one gigantic format bump
the existing ctf will always be readable
even when new formats are
even when new formats are generated um i
don't see any reason to ever drop
support for old versions
um and there are more things there's a
little there's a link here to them
anyway actually
um
let's
then
kick off the discussion on bad traces by
looking at some use cases and how their
requirements vary
i have classified them into two
categories online and offline the
category of online back tracing is
basically all those
use cases where the application is in a
production like environment and the cost
of generating bad traces is borne by the
application the cost being um cpu cycles
and program memory because the relevant
debug sections need to be loaded in the
memory
conversely in case of offline back
tracing the cost of back tracing is not
draining on the application as they are
often run after the fact like in a crash
report analysis tool or a debugger
so classic examples of online back
tracers are profilers crash reporters
exception
handling and live patching
now let's look at the requirements one
by one
online back tracers need to be fast
right they need to be precise um
sometimes extremely precise like in case
of life patching they need to be
extremely precise otherwise they are
well
it's not usable otherwise
asynchronous unwind it's a capability
which basically means that you need to
be able to unwind uh from any pc now
that capability is also a must-have for
online back tracers um
to achieve all these goals it is vital
that the unwinder itself be small and
that the debug information for unwinding
itself as well be small right so
so
both so online back tracers need that
the unwinder be small and the
debug info as well be small some online
back tracers may also need
uh the original value of the arguments
in back traces
uh in contrast now for offline back
tracing use cases all these features are
in a good to have category as you can
reason about them so the takeaway
from this slide is that there are these
two categories of back tracing and the
requirements vary a lot
and we are focusing on the case of
online back pricing um for
cdf
so for online back tracing what is the
state of the art how are these generated
anyway right so in many if not most
cases online back tracing is based on eh
frame sections um
these are dwarf based sections
and these sections encode the
fundamental
information needed for back tracing
which is given a pc what is the cfa and
ra cfa is a short form for canonical
frame address uh think of it as the
frame address which is
um
which is at the before the function
starts what is the frame address and see
if it just points you to that
each frame sections are the default for
many targets
and um there is this eh frame header
sections as well which are basically
which basically allow you a faster look
up of the information in each frame
so using read elf let's see the contents
of
a sample eh frame section as you see
each frame gives us clearly for each pc
a way to recover cfa ra and some other
quality registers if they made it if
they made it to stack um
so
as you also see sometimes if you go from
one pc to the next there is not much
change in some columns so look at rbp in
the first ip it is undefined but there
onwards for each ip it is defined but it
is the same so it says cfa minus 16 it's
present rpp is stored at a location cfa
minus 16. now
so there is a lot of repetition in this
information right if stored in this
tabular format
and this is precisely the reason what
each frame actually stores is some dwarf
opcodes
which are executed on a simple stack
machine to generate the contents of the
table we saw in the previous slide so
this is a compact way of storing that
kind of information but if you are the
unwinder you need to execute these
opcodes
at runtime to figure out what this how
the cfa recovery will be done right
so
that creates a problem
for
for any unwinders written for for
basically eh frame based unwinders
and
they be they are slow because at the
time of this this makes them slow
because at the time of uh generating the
bad traces you now need to execute these
opcodes to figure out the cfa and ra etc
they are large because of the same
reason the unwinders need to implement a
simple stack machine
to
exec to to run these opcodes and figure
out the information at runtime now the
second issue with each frame based
approach is the maintenance of
handwritten assembly now um
so these these eh frames sections they
are generated by the assembler and the
simpler processes what are called the
cfi directives and if you hand code
parts of your application then you as a
programmer are also responsible for hand
coding these cfi directives as well cfi
are call frame
information directives and this can
sometimes get quite complicated because
if you're if your standard an assembly
uses stack in a standard way this is
relative this is arguably easy to do
but if your handwritten assembly does
non-standard stack usage then you may
have to deal with what are called dwarf
expressions now writing cfi directives
with dwarf expression is not going to be
very maintainable
lastly if you
if you do end up needing original value
of your arguments for your online back
tracing that's not going to be easy the
information is stored in dwarf debug
sections and which can be pretty large
to consume at runtime
so
in reality because of these issues with
eh frame based methods there exists a
number of unwind formats so that's the
state of the art right each frame is not
the only one out in the field
uh
many applications that we have looked
around um
they have tried to come up with their
own debug formats trying to
you know steer away from eh frame based
methods kernel for example has its own
unwind format called the
object rewind capability
no sorry oops rewind capability perhaps
anyway various other projects that we
ran that we ran into also created their
own ad hoc solutions around the problem
uh the important bit here is that these
ad hoc solutions and their unwind
formats are not supported in the no tool
chain and which makes all of these uh
addox solutions a big ordeal
and lastly well if you still want to
recover if you still need means to
recover original value of the arguments
the state of the art remains that
you need to get that information from
dwarf call site information which lies
in the debug info sections now remember
all those problems with dwarf they mean
the dwarf may be stripped out they are
large and it's going to be slow just
because of the size of the things
so
cdf backtrace format aims to solve
these issues by providing um a solution
um it tries to it the the premise here
is that if you aim for the following
requirements you will be able to solve
uh the previously mentioned problems
number one
you must provide means for asynchronous
track unwinding otherwise the back
traces are not going to be useful number
two
yes
have some extensions in the ctf format
but keep the format compact and simple
do not add complex expressions encoding
no stack machines whatsoever
and number three allow means to recover
original value of the arguments to the
function call
some online back racers need this and
it's a must have for them
excuse me
so
we plan to introduce a new section
called
dot ctf frame
we will only talk about high level
details at this time
the section stores information on how to
recover
cfa and ra given the pc
it does not store information on how to
recover other quality saved registers
conceptually it's going to be equivalent
right to the table of interpreted dwarf
data
but not the full table as compared to
each frame because we only need to know
about cfa and ra right so we are
starting out with uh
support for amd 64 and air 64 but there
is no reason why other
arches and abis cannot be accommodated
we are working to support for the
generation of cdf batteries format in
the new tool chain uh toolchain is the
ideal place where such a format
needs to be supported because we know
from existing ad hoc solutions that
anything other than tool chain becomes
very difficult to maintain and extend
so let's talk about what cdf backtrace
format has to offer for recovering
original value of the arguments
in debug format jargon this is often
called call site information uh let's
first revisit the background again
how are arguments passed to a function
right so there is the there is these
abi's that define the parameter passing
rules um the first bit that an abi
specifies as how to assign
a storage class to an argument so given
a high level language type if you write
foo and it has two arguments in a
in b
it tells you so given these high level
language types inc and floats and
whatnot what is the corresponding
machine type
aka the storage class next
excuse me given the storage class the
abi rules the api defines rules on how
to define how to assign
appropriate register or stack location
excuse me
so the advantage of uh of of of
approaching the problem in this way is
that you can
you can potentially find a way to
relieve the debug format of specifying
the exact register location
right so
in this case um the exact register
location set is going to be a larger set
whereas storage location set is going to
be a smaller set across apis now each
api as you know will offer you know
different set of registers and
but the storage classes are going to be
pretty similar across avis or a smaller
set rather right
so uh as an example let's see amd 64
looks like this integer ssc sse
up memory and so on
for ar 64 similar looking classes with
some differences of course right so as
you see the set of storage classes and
amd 64 look similar so this is the main
premise we are relying on
the storage location of the argument is
not encoded explicitly in the debug
format the natural location is inferred
from the position and class of the
argument and the abi
by the client right it's not encoded in
the debug format so
if it is a bit hazy still i hope it is
clearer with this example
so function one is a function with six
integer arguments
the amd64 avi mandates that the
arguments be passed via rdi rsi
rcx and so on in order
now as far as the call side information
is concerned it should be enough if the
debug format conveys that the arguments
are of integer class
and the backtrace client with an
understanding of the api can conclude
that
because the storage class was integer i
know that these
arguments will be passed via rdi rsi and
so on so in interest of time again i
will skip through the second example
where one of the arguments is a small
aggregate
but as you see what we need is just an
unambiguous way to convey the storage
class
and the rest can be done by the
batteries client
so the call site information edition is
being planned in two phases so that the
addition of the cdf format the additions
to the cdf format are done in a
data driven and thoughtful way in phase
one we will focus on what we talked
about previously that the original value
of the argument can be recovered from
its natural location if it is not
clobbered and what you encode is just
some information about storage classes
now clearly there will be cases when
these natural locations will be
clobbered by the colleagues so an option
so as an option there is to
there is to fall back on
is
is explicit saving of argument values on
stack if the user chooses to do so and
the format facilitates the recovery
to
of the original value of the arguments
from stack
so
to summarize ctf back traces um we
discussed today that the format
extensions
uh
to support online information and call
site information and underway
we looked at the requirements of online
back tracing and to fulfill those
requirements we have to keep the format
minimal
um
we are working on these format changes
and incorporating the support in gnu
tool chain
finally to wrap it all up
ctf we introduced ctf
today and a cdf is a debug format that's
compact and fast
at this time it is
type information for c programs only and
is fully supported in the gnu tool chain
it is evolving to address unfulfilled
requirements around debugging and back
tracing if you are interested and if you
want to participate have questions you
can reach us on any of these
public mailing lists
thank you
thank you so much indo uh and nick and
edsentia for um for sharing all that
with us wow um that was
really interesting and thorough look at
i guess the new possibilities um in the
future so
thanks um
is there anything you'd like to say
thanks
[Music]
well yes thank you so much round of
applause for indo and nick um
and that was our last talk for today
everyone last talk for
linuxconfigu2022 but don't close the
window yet um so we will have the
conference closing in this very room um
in
about 15 minutes because this talk is
finishing up a little early
so don't go anywhere conference closing
is always worth watching um it'll be
lovely to see the wrap up
um but for me that's it from me so thank
you for joining us all today with all of
these wonderful talks
bye everyone
you