How to catch/grabb all links?

Off-topic posts of interest to the "Everything" community.
Post Reply
Debugger
Posts: 531
Joined: Thu Jan 26, 2017 11:56 am

How to catch/grabb all links?

Post by Debugger » Sun Dec 16, 2018 10:43 am

How to catch all links? (Not to be confused with downloading) I used to have a browser add-on, now I do not know what the add-on is suitable for.

Here from last week's music:
https://soundcloud.com/search/sounds?q= ... =last_week

therube
Posts: 2377
Joined: Thu Sep 03, 2009 6:48 pm

Re: How to catch/grabb all links?

Post by therube » Sun Dec 16, 2018 12:16 pm

See if something here does what you need, Link Bookmarklets?

Debugger
Posts: 531
Joined: Thu Jan 26, 2017 11:56 am

Re: How to catch/grabb all links?

Post by Debugger » Sun Dec 16, 2018 1:08 pm

therube wrote:
Sun Dec 16, 2018 12:16 pm
See if something here does what you need, Link Bookmarklets?
There is nothing here that can catch some or most of the links. The application must process JavaScript (!).
need a proper script to process the links, but do not scroll the page down, which is a damn time-consuming,
I need something like Extract the links on this page

therube
Posts: 2377
Joined: Thu Sep 03, 2009 6:48 pm

Re: How to catch/grabb all links?

Post by therube » Sat Apr 20, 2019 11:59 am

If you were looking for the song URL, (in a Mozilla browser) you could open Developer Tools -> Network, & filter on .m3u8.

Actually you don't even need to "play" any of them.
So long as you focus (hover) an item, it's URL will show up.

With the URL, you can open it externally, say in MPlayer.

Code: Select all

https://cf-hls-media.sndcdn.com/playlist/NX82UjJgAgGK.128.mp3/playlist.m3u8?Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiKjovL2NmLWhscy1tZWRpYS5zbmRjZG4uY29tL3BsYXlsaXN0L05YODJVakpnQWdHSy4xMjgubXAzL3BsYXlsaXN0Lm0zdTgiLCJDb25kaXRpb24iOnsiRGF0ZUxlc3NUaGFuIjp7IkFXUzpFcG9jaFRpbWUiOjE1NTU3NjE5MjN9fX1dfQ__&Signature=kjXgnwjBwiL35lEmYXQb4mV3RdRVq3MsQk89u1NXjl-34fKiyytYVBroSrFWk-pKvEMUQBu7HE-g-S2Fu40itdsdQDY-XnT4G2wtz3huf2xurRnm1CRQi6yk81y8Xc~V7Fo1XNC-kKYC~oHCHmqjmP4Drtz~nYiPYu5WfWL8QRmg3wlDGxKcP6feIeT8V8QyaIddV8NHDIotu76hQP~sdxXamZdS5p88XBTOvl-talB8ikkzqsYPrIrttaNdxoR3IJwJ0RjDhJ~iM4EtU92kkS4dP--AJzrdYibNtMUmQq1uvszl-S3HZ-IiGOeJXUUahirFv1DQyoLuuA5Kitw3BA__&Key-Pair-Id=APKAJAGZ7VMH2PFPW6UQ
---

Bulk Media Downloader (a sniffer) will not find those links.

Debugger
Posts: 531
Joined: Thu Jan 26, 2017 11:56 am

Re: How to catch/grabb all links?

Post by Debugger » Sat Apr 20, 2019 3:55 pm

therube -
I only need a whole list of source links:

Example:
https://soundcloud.com/user-290759768-1 ... kina-remix
https://soundcloud.com/djparadox79/dj-p ... s-monta-06

Unfortunately, without processing javascript you can not extract the subpage, you have to load it manually.
Well, unless there is an extension that can do it, but there is no such extension anywhere!


Image

therube
Posts: 2377
Joined: Thu Sep 03, 2009 6:48 pm

Re: How to catch/grabb all links?

Post by therube » Sun Apr 21, 2019 12:20 pm

It would seem (in this case) "links" are "generated on the fly" - rather then static, likely have a limited lifespan, & may or may not be accessible to others (such as the link I posted above, is now "dead").

Now maybe you could write some code, the same as the page uses, that iterates through the files shown on the page, generating valid URL to files?

https://www.google.com/search?q=firefox ... +extension
I would be CAREFUL with any of these types of extension found on addons.mozilla.org.
I would be careful, in general, with any extension found on addons.mozilla.org.

Debugger
Posts: 531
Joined: Thu Jan 26, 2017 11:56 am

Re: How to catch/grabb all links?

Post by Debugger » Sun Apr 21, 2019 12:45 pm

therube

First. How wrong you are. The source links are online and play and in no case change. (The material has not been deleted!). Well, unless you mean something else?

Secondly - when it comes to soundcloud.com extensions, they are primarily aimed at downloading files. And that's not what I mean.
But, well - there is an extension to "catching(grabb) all source links", but as I said before, to process the whole page, scroll down the page and then "catch(grabb) all". It's such a fun, but it consumes a lot of time!

therube
Posts: 2377
Joined: Thu Sep 03, 2009 6:48 pm

Re: How to catch/grabb all links?

Post by therube » Mon Apr 22, 2019 12:03 pm

I meant that a link, once "generated", only has a limited lifespan.
The physical file remains, but the link (URL) to that file will differ over time (& also may be "unique" to a particular "user").

Debugger
Posts: 531
Joined: Thu Jan 26, 2017 11:56 am

Re: How to catch/grabb all links?

Post by Debugger » Mon Apr 22, 2019 12:48 pm

therube - Yes, but you must distinguish the type of address, e.g.
Referrer, Source or Data. There are others, but these are the most important ones.

Direct links (DATA) (generated) usually have a limited time (session expiration time, or assigned to your IP). Such files can be downloaded immediately.

------------------------------------
Source.
So they do not change. And I am talking about such links :)
viewforum.php?f=7

Post Reply