KnowledgeHierarchy

来自集智百科 - 复杂系统|人工智能|复杂科学|复杂网络|自组织
13944556914讨论 | 贡献2024年5月21日 (二) 15:50的版本 →‎References
(差异) ←上一版本 | 最后版本 (差异) | 下一版本→ (差异)
跳到导航 跳到搜索

RQ: what is a good question

Zhihu tag tree

知乎数据抓取 https://github.com/7sDream/zhihu-oauth

https://www.zhihu.com/topic/19776749/organize/entire#anchor-children-topic

import pandas as pd

# https://github.com/Lynxmac/zhihu_topic_tree/
with open('/Users/datalab/bigdata/zhihu_topic_tree.txt', 'r', encoding='gb18030') as f:
    lines = f.readlines()

df_list = []
for index, line in enumerate(lines):
    a = line.rstrip().split('─')
    hierarchy = len(a[0])
    if index > 312482:
        hierarchy -= 1
    sign = a[0][-1]
    b = a[-1].split('_', maxsplit = 2)
    ids = b[0]
    name = '_'.join(b[1:])
    df_list.append([index, hierarchy, sign, ids, name, line])

df = pd.DataFrame(df_list, columns = ['loc', 'hierarchy', 'sign', 'id', 'name', 'line'])

# clean the hierarchy variable
new_hierarchy = []
for i in df.hierarchy:
    if i % 3 ==1:
        new_hierarchy.append(i)
    elif i%3 ==2:
        new_hierarchy.append(i-1)
    elif i%3 ==0:
        new_hierarchy.append(i-2)
        
df['new_hierarchy'] = new_hierarchy
df['good_hierarchy'] = [(i-1)/3 + 1 for i in new_hierarchy]

# add missing id for level 1 topics
id_list = [(29855, 19778298, '「形而上」话题'), 
 (178555,19560891,'产业'),
 (190122, 19618774, '学科'),
 (223661, 19778287, '实体'),
 (312482, 19778317,'生活、艺术、文化与活动')]

for i in id_list:
    df['id'][i[0]] = i[1]
    df['name'][i[0]] = i[2]

# delete wrong ids
error_id_index = []
for k, i in enumerate(df.id):
    try:
        j = int(i)
    except:
        error_id_index.append(k)
len(error_id_index)

df = df.drop(error_id_index)
df['id'] = [int(i) for i in df.id]

# construct network
# it takes around 3 hours
# search for the nearest high level neighbor and link together

from flownetwork.flownetwork import flushPrint

net = []
for i in df.index:
    if i%100 ==0:
        flushPrint(i)
    ids = df['id'][i]
    hierarchy = df['good_hierarchy'][i]
    loc = df['loc'][i]
    if hierarchy == 1:
        net.append(('root', ids))
    else:
        upper_hierarchy = hierarchy - 1
        upper_nodes = df[df['good_hierarchy'] == upper_hierarchy]
        upper_node_loc = [j for j in upper_nodes['loc'] if (loc - j) > 0][-1]
        upper_node_id = df['id'][df['loc'] == upper_node_loc]
        net.append(( int(upper_node_id), ids))

StackOverflow tag network

StackOverflow using tags to organize raised questions, see the tags here: https://stackoverflow.com/tags

Given a tag, such as javascript, you can see the tagged questions: https://stackoverflow.com/questions/tagged/javascript

Note that, stackoverflow also demonstrates the related tags for a tag. For example, the javascript tag is related to

Related Tags

  • jquery × 518122
  • html × 317264
  • css × 146448
  • angularjs × 116430
  • php × 111251
  • node.js × 92095
  • ajax × 88493
  • json × 58117
  • html5 × 51808
  • reactjs × 51308
  • arrays × 49362
  • asp.net × 31550
  • regex × 28362
  • twitter-bootstrap × 24516
  • angular × 24174
  • c# × 23339
  • forms × 22346
  • google-chrome × 21292
  • d3.js × 21232
  • dom × 19442
  • google-maps × 18658
  • typescript × 18244
  • java × 17724
  • canvas × 17054
  • express × 16103

Quora

https://www.quora.com/topic/Computer-Science

https://github.com/tapaswenipathak/pyQTopic/blob/master/qtopic/pyqtopics.py

https://github.com/csu/quora-api

References

User:Wangchj04