UCSC custom track upload direct data POST / without url

Question

[San]

Hello All,

Documentation seems rather scarce on this topic so I just come here to find a consensus. In some software I write for the lab I'm in, there is a feature to view on UCSC. This feature will create a small BED file (usually ~10-20 lines), upload to UCSC with a post request, parse the returned data for the "hgsid" , and redirect user to the UCSC page with the proper hgsid parameter to display the custom tracks. Here is an example of a script which does this in python:

from io import StringIO

import re
import requests

bed_file = """browser position chr18:61799395-61805146
browser hide all
track name="Custom Track" description="Generated by custom track export" visibility=2 itemRgb="On" useScore=1
chr18 61799395 61805146 Exons 0 - 61799395 61805146 127,127,127 2 2013,80 0,5671
chr18 61801409 61805065 De_Novo_Introns 0 - 61801409 61805065 94,182,91 1 3656 0
chr18 61801408 61805066 Annotated_Junctions 0 - 61801408 61805066 230,52,54 1 3658 0
"""

tmp_file = StringIO()
tmp_file.write(bed_file)
tmp_file.seek(0)

files = {'hgct_customText': tmp_file}
payload = {'db': 'mm10'}

r = requests.post('http://genome.ucsc.edu/cgi-bin/hgCustom', files=files, data=payload)

content = r.text

hgsid = re.findall(r'hgsid=[0-9]*_[a-zA-Z0-9]*', content)[0].split('=')[1]
link = f'http://genome.ucsc.edu/cgi-bin/hgTracks?hgsid={hgsid}&position=chr18%3A61799395-61805146'

print(link)

I am writing because it seems like is seems this method has stopped working silently. The POST seems to submit and not raise an error, and I can still parse the hgsid from the result, however, the custom tracks are not shown after following the link. Can anyone confirm there have been some minor tweaks to the API which I need to change, or whether UCSC has put an announcement somewhere that this feature has been discontinued? I'm having trouble finding official documentation on it now, but I know for sure it used to work less than a year ago.

Thanks.

[Pierre Lindenbaum]

I'm not sure there is an official way to scrap the UCSC form. A safer way would be to create and update a custom UCSC hub

[San]

Hi Pierre,

I'm not sure what you mean by "scrap the UCSC form". In this case there seem(ed) to be a supported parameter to the form "hgct_customText" which would just submit the custom track with that data. It wasn't intended to be a hack but rather a supported api input field.

What is a "custom UCSC hub", do you know if it has any similar functionality? The custom tracks are very ephemeral , similar to the regular custom track upload form, not intended to be kept permanently.

[Pierre Lindenbaum]

In this case there seem(ed) to be a supported parameter to the form "hgct_customText" which would just submit the custom track with that data.

ok, I don't know that API. Where is the doc please ?

What is a "custom UCSC hub",

https://genome.ucsc.edu/goldenPath/help/hgTrackHubHelp.html

[San]

Hi Pierre,

I can only assume that the feature muse have been made scarce for some reason. The closest official post I see is here: https://genome.ucsc.edu/goldenPath/help/customTrack.html under "example #6" Where it mentions "the URL must include an hgct_customText parameter, which simulates the text box on the Custom Tracks page." however, this example uses the hgct_customText along with a separate hosted data url, whereas I'm trying to use hgct_customText only.

It doesn't appear that I can use the custom UCSC hub solution as this requires some external web server set up. I was more looking for a way to take a few track lines, and give a user a way to click a button in a program which opens UCSC with those small tracks added. It appears now that the only method is to copy the bed file text to clipboard and have the user paste it into the custom track page instead, which is an extra few steps, still doable just annoying for users. I am wondering why the old better method has stopped working.

Thanks.

[Pierre Lindenbaum]

s "the URL must include an hgct_customText parameter, which simulates the text box o

example 6 says: " Let's assume that your data is on a server at your institution"

[Maximilian Haeussler]

As you can see from the other reply, the problem is more related to the type of the redirect and this particular Python (?) library. We have not made uploads harder. It's true that we prefer and recommend track hubs, but we can't phase out custom tracks, too many websites use them.

Track hubs have many more options and are not too hard to set up. They are stored on your server, not ours, so you have much more control over the data. You have a ton more options, colors, mouseovers, container tracks, etc. We're not going to add more features to custom track but are constantly improving track hubs.

Answer 1

[Maximilian Haeussler]

We have not changed this form. I wouldn't call it an "API", but yes, it's similar, this has been possible for a long time and was documented. We haven't changed anything about this, but I have a feeling that this could have to do with our http -> https redirects that were activated around 2 weeks ago. A redirect converts a POST to GET request in some cases, I think, I'm blanking on this now, but I think some R libraries have problems with that, maybe Python, too? Can you try the same thing, but with https://genome.ucsc.edu instead of http://genome.ucsc.edu and see if that changes something?

[San]

Hi Maximilian,

Thank you very much for the reply! It really was as simple as switching from http:// to https:// in the two hardcoded URLs. I really appreciate that I'll now be able to include this functionality in the next version of the MAJIQ splicing analysis software. ^.^