The Twitter Timeline

Peter Smyth

Diced project data manager

UK Data Service Webinar

20 April 2021

What we will look at

1. Keys & Authentication

2. The new API and developers account

3. What is and what isn't in a users Timeline

4. Extracting a complete timeline ,

while avoiding the API rate limits

5. Stitching things together

6. Conversations

7. Analysis of the timeline contents.

Keys & Authentication

• API v2 now uses a ‘Bearer Token’ for access

• Overall this makes things easier

• But you still need to get one before you start!

• And to get one you need a Twitter account!

The APIs

• Currently v1.1 and v2

• V1.1 well established

• V2 still being developed, but simpler to use

• We will assume that any one new to the Twitter

API

would choose v2

• This is a non-t

echnical Webinar, so for today it

doesn’t matter

The Old System (v1.1)

• You can however still use it if you have the right

set of keys

• You can also use it with some v2 keys

• If you have old keys and legacy code, will still

ork

• No talk about when it will be shut down - t

hat I've

seen!

The new system

• Dashboard!

• Changes to endpoints

• Changes to data delivered

• Instead of getting everything

• You now have to say what data you want returned

Old System returned data

Academic accounts

• 10,000,000 Tweets / month

• Access to Full Search

• Need to apply and be accepted

Academic Accounts – Application process

• Make sure you have a Twitter Account before

you start

• Complete relatively short online form

• Wait a day or so and hope

• Not sure what actual criteria are, but you are

ked about You and Your project

Academic Accounts – Application process

• Online form (Edge)

Timelines v What do you see

Four things to consider:

1.What is included

2.What is not included

3.What is not included but you can get

4.What is not included and you cannot get

Included in what you see

• Your own Tweets

• Tweets from those you follow (your friends)

• Twitter recommendations

• Adverts

• Tweet Warnings

Not included (in what you see)

• All the tweets your friends see

• Their recommendations and Ads

Timeline – What you can get

• All of your Tweets

• All of your Friends Tweets

Timeline – What you cannot get

• Deleted Tweets

• Anything from banned accounts

• Adverts

• Recommendations (from Twitter)

Example Twitter Feed

Software choices (Python based)

1.Python Tweepy

2.Python twa

3.TwitteR

4.Raw API (via Python Requests)

Plus

many more possibilities

Example : The UKDS Timeline

• Basic Stats:

• 3248 Tweets

• Earliest date is : 2018-06-05T14:32:34.000Z

(Tweet rate of 3.1/day)

Example : The UKDS Timeline

Made up of:

691 Original Tweets

2296 Re-Tweets

103 Quoted Tweets

158 Replies

Significance of the Tweet Rate

• The higher the Tweet rate, the shorter the

Timespan of in the Timeline (limited to 3200 most

recent Tweets)

• May be as short as a couple of weeks

• Need to collect them promptly (assuming you

an)

Significance of the Tweet Rate

• Example: DiCED

• Want 905 complete Timelines

• Total of 300030 unique Friends Timelines

• Took over 16 days to collect

Significance of the Tweet Rate

• Situation easier with Academic account as you

have access to full search and can specify start

and end times

Collecting the data

We need:

1. The UKDS Timeline

2. A list of the UKDS Friends

3. The Timelines of all of UKDS Friends

Alt

hough not part of the timeline, it is useful to

have User details of the UKDS and their Friends

(UKDS_friends.csv)

Example: The UKDS User Info

"data": {

"created_at": "2011-09-16T08:10:34.000Z",

"description": "The UK\u2019s largest collection of UK and international social,

economic and population data.

Funded by @ESRC. Visit our blog: https://t.co/EBCD4NqGKW",

"id": "374406950",

"name": "UKDataService",

"public_metrics": {

"followers_count": 10137,

"following_count": 3356,

"listed_count": 378,

"tweet_count": 10535

"url": "http://t.co/GUYNj9RsnE",

"username": "UKDataService",

"verified": false

}

A word about Trump

• Controversial Tweets

• Free Speech

• Warning on Tweets

• Opinion of Fact checker or Twitter

• Tweets Removed

• No longer returned in searches

• Donald J Trump Banned!

• He becomes a non-person to Twitter

Non-Person on Twitter

• Not only can you not retrieve his Tweets

• His Id is removed from the Friends list of his 88M

ollowers

What data to keep

• In v2 of the API you make this decision in

advance by what you ask for

• In v1.1 as you get everything, you may decide to

get

rid of the junk, but there will be key fields you

want to keep

Field Essential Likely

lang

text Y

author_id Y

id Y

created_at Y

conversation_id Y

possibly_sensitive

source

context_annotations

referenced_tweets Y

reply_settings

entities.mentions Y

entities.hashtags Y

public_metrics.retweet_count Y

public_metrics.reply_count Y

public_metrics.like_count Y

public_metrics.quote_count Y

entities.annotations

entities.urls Y

attachments.media_keys

in_reply_to_user_id Y

attachments.poll_ids

What to keep (Twitter_Fields.xlsx)

Timeline Reconstruction

1. Get the Target Timeline (UKDS)

2. Get List of Target Friends

3. Get Timelines from Friends

4. Combine ‘Data’ and ‘Includes sections of

imeline Data

5. Combine (append) all of the Timelines

6. Sort by Date or ID

4. Combine ‘Data’ and ‘Includes sections of

Timeline Data

• If the Tweet was a Re-tweet, then you need to

data in the ‘Includes’ section to get the full text of

the Tweet.

• The same is true for a complete set of Entities

(

‘hashtags, @manetions, URLs )

What Next?

• The Timeline itself is just a Chronological

sequence of events

• How you use it is upt

o you, but probably

influenced by your research needs

Conversations and Responses

• If you are going to do any kind of Text analysis,

you will probably need to combine Tweets into

conversations.

• Or even just re-c

onstruct multi-part Tweets

• API v2 helps here by providing a ‘Conversation’

end

-point which will provide all of the Tweets with

the same Conversation_id (You still have to

assemble them yourself)

Timeline Analysis

• What do you want to do?

• Have you kept the data to do it?

• Do you need additional software or Data?

Timeline Analysis

Examples:

• Change of Tweet rate

• Trending (external) Hashtags

• Retweeting new/different sources

Sentiment Analysis

• Tricky with limited text

• Some of which is hashtags, mentions and URLs

• Emojis!

• Irony and sarcasm – N

• (small demo)

Basic Stats (and other things)

• How many Original Tweets v Retweets

• User info. – H

ow old account, No. of Tweets,

Friends and Followers

• User Description

• Tweet rate over time

• Relate @mentions and #hashtags to known

rends or events

Examples of Hashtag usage

Warning about shortened URLs

[{'start': 105,

'end': 128,

'url': 'https://t.co/KhiIOOPzVO',

'expanded_url':

'https://www.theyworkforyou.com/wrans/?id=2021-03-

24.HL14586.h',

'display_url': 'theyworkforyou.com/wrans/?id=2021…'

}]

Network Analysis

• Often Friends & Followers

• Could be accounts and #hashtags or @mentions

• Many additional s/w products support network

aphs.

• Social Networks is a subject in its own right.

• (small demo)

Twitter producers and consumers

• Friends and Family

• Hobbies and pastimes

• Commercial/Public services

• News Outlets

• Political Organisations

Tweet Traffic

Summary

• Have a Plan

• Re-us

ing a plan could be as simple as changing

the account name

• The Timeline by itself is just data – y

ou have to

turn it into information

• There is much pre-built

s/w to help with the

techie bits.