信息检索领域相关资料 (A Guide to Information Retrieval)
Organized by Hongfei Yan
Last updated on July 27, 2007

---------------------
Contents
	Books
		+ Finding Out About: Search Engine Technology from a cognitive 
			Perspective (Belew, R.K., 2000)
			http://www-cse.ucsd.edu/~rik/foa/
		+ Foundations of Statistical Natural (C. Manning and H. Schutze, 1999)
		+ Information Retrieval, 2nd edition (C.J. van Rijsbergen, 1979)
			(full text)
			http://www.dcs.gla.ac.uk/Keith/Preface.html
		+ Information Retrieval: A Survey (Ed Greengrass, 2000)
			http://www.csee.umbc.edu/cadip/readings/IR.report.120600.book.pdf
		+ Information Retrieval: Data Structures & Algorithms
			(Frakes, W. and Baeza-Yates, R., 1992)
			http://www.dcc.uchile.cl/~rbaeza/iradsbook/irbook.html
		+ Information Retrieval Interaction (Ingwersen, P., Taylor Graham, 1992)
			http://www.db.dk/pi/iri/
		+ Introduction to Information Retrieval
			(Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schuetze, 2007)
			http://www-csli.stanford.edu/~schuetze/information-retrieval-book.html
		+ Managing Gigabytes:compressing and indexing documents and images,
			2nd edition, (Ian H. Witten, Alistair Moffat,and Timothy Bell,1999)
		+ Mining the Web: Discovering Knowledge from Hypertext Data 
			(Soumen Chakrabarti, 2003)
		+ Modeling the Internet and the Web: 
			probabilistic Methods and Algorithms 
			(Pierre Baldi, Paolo Frasconi and Padhraic Smyth, 2003)
		+ Modern Information Retrieval 
			(Ricardo Baeza-Yates and Berthier Ribeiro-Neto, 2000)
		+ Readings in Information Retrieval. 
			(Sparck-Jones, K. and Willett, P., 1997)
		+ Search Engine: Principle,Technology and Systems 
			搜索引擎-原理、技术与系统
			(Xiaoming Li,et al., 2005 ), (full text)
			http://sewm.pku.edu.cn/book/dlbook.html
		+ The Geometry of Information Retrieval 
			(C.J. van Rijsbergen, 2004)
			http://ir.dcs.gla.ac.uk/GeometryOfIR/
		+ The Turn: Integration of Information Seeking and Retrieval in Context
			(Ingwersen, P., and Jarvelin, K., 2005)
		+ TREC: Experiment and Evaluation in Information Retrieval 
			(Voorhees, E.M., and Harman, D.K., 2005)
			http://mitpress.mit.edu/catalog/item/default.asp?ttype=2&tid=10667

	Conferences and Workshops
		+ CIKM: Conference on Information and Knowledge Management
			http://www.csee.umbc.edu/cikm/
		+ SIGIR: Special Interest Group on Information Retrieval
			http://www.sigir.org/
		+ SIGKDD: Knowledge Discovery and Data Mining
			http://www.kdd.org/
		+ World Wide Web
			http://www.iw3c2.org/
		+ SEWM: Symposium of Search Engine and WebMining
			全国搜索引擎和网上信息挖掘学术研讨会
			http://net.pku.edu.cn/~sewm/

	Courses
		+ CMU Information Retrieval
			http://nyc.lti.cs.cmu.edu/classes/11-741/ (Spring 2006)
			Instructors: Jamie Callan and Yiming Yang 
		+ Cornell University The Structure of Information Networks (Spring 2006)
			http://www.cs.cornell.edu/courses/cs685/2006sp/
			Instructor: Jon Kleinberg
		+ Peking University Web Based Information Architectures (Fall 2006)
			http://net.pku.edu.cn/~wbia/
			Instructor: Xiaoming Li, Jimin Wang and Bo Peng
		+ Stanford Univ. Text Information Retrieval and Web Mining (Autumn 2005)
			http://www.stanford.edu/class/cs276/
			Instructor: Christopher Manning and Prabhakar Raghavan
		+ UIUC Introduction to Text Information Systems (Spring 2007)
			http://sifaka.cs.uiuc.edu/course/410s07/
			Instructor: ChengXiang Zhai
		+ UMass Univ. Information retrieval course (Spring 2005)
			http://ciir.cs.umass.edu/cmpsci646/
			Instructors: James Allan
		+ Washington Univ. Search Engines course
			http://courses.washington.edu/lis544/

	Evaluation Resources
		+ CLEF: Cross-Language Evaluation Forum
			http://clef.iei.pi.cnr.it/
		+ CWIRF: Chinese Web Information Retrieval Forum
			http://www.cwirf.org/
		+ DUC: Document Understanding Conferences
			http://duc.nist.gov/
		+ INEX: INitiative for the Evaluation of XML Retrieval
			http://inex.is.informatik.uni-duisburg.de/
		+ NTCIR: NII-NACSIS Test Collection for IR Systems
			http://research.nii.ac.jp/ntcir/
		+ TREC: Text REtrieval Conference 
			http://trec.nist.gov/

	Journals
		+ Briefings in Bioinformatics (full text)
			http://bib.oxfordjournals.org/archive/
		+ Computational Linguistics, The MIT Press
			http://mitpress.mit.edu/catalog/item/default.asp?ttype=4&tid=10
		+ Data & Knowledge Engineering (DKE), Elsevier
			http://www.elsevier.com/wps/find/journaldescription.cws_home/505608/description?navopenmenu=-2
		+ D-Lib Magazine
			http://www.dlib.org/
		+ Information Processing Letters, Elsevier
			http://www.elsevier.com/locate/issn/00200190
		+ Information Processing and Management (IP&M), Elsevier
			http://www.elsevier.com/locate/infoproman
		+ Information Retrieval, Springer
			http://www.springer.com/sgw/cda/frontpage/0,11855,3-0-70-35744790-detailsPage%253Djournal%257Cdescription%257Cdescription,00.html
		+ Information Research
			http://informationr.net/ir
		+ International Journal on Digital Libraries, Springer
			http://link.springer.de/link/service/journals/00799/index.htm
		+ International Journal of Cooperative Information Systems (IJCIS), 
			World Scientific
			http://ejournals.wspc.com.sg/ijcis/ijcis.shtml
		+ International Journal on Document Analysis and Recognition, Springer
			http://link.springer.de/link/service/journals/10032/index.htm
		+ International Journal of Intelligent Systems, Wiley
			http://www3.interscience.wiley.com/cgi-bin/jhome/36062
		+ International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems (IJUFKS), World Scientific
			http://ejournals.wspc.com.sg/ijufks/ijufks.shtml
		+ Journal of the American Society for Information Science and Technology (JASIST), Wiley
			http://www3.interscience.wiley.com/cgi-bin/jhome/76501873
		+ Journal of Documentation (JDoc). Emerald
			http://www.emeraldinsight.com/0022-0418.htm
		+ Journal of Intelligent Information Systems (JIIS), Springer
			http://www.wkap.nl/journalhome.htm/0925-9902
		+ Knowledge and Information Systems (KAIS), Springer
			http://link.springer.de/link/service/journals/10115/index.htm
		+ Natural Language Engineering, Cambridge University Press
			http://www.cambridge.org/journals/journal_catalogue.asp?mnemonic=NLE
		+ Transactions On Information Systems (TOIS), ACM
			http://www.acm.org/tois/
		+ Transactions on Knowledge and Data Engineering (TKDE), IEEE 
			http://www.computer.org/tkde/

	List Archives
		+ SIG-IRList, http://www.sigir.org/sigirlist/index.html

	Organizations and Special Interest Groups
		+ Cambridge NLIP, http://www.cl.cam.ac.uk/Research/NL/
		+ CMU LTI, http://www.lti.cs.cmu.edu/
		+ DEC laboratories in Palo Alto, Calif.
		+ Glasgow Information Retrieval Group, http://www.dcs.gla.ac.uk/ir/
		+ Google Labs, http://labs.google.com/
		+ LTI, http://www.lti.cs.cmu.edu/
		+ Massachusetts CIIR, http://ciir.cs.umass.edu/
		+ MSR Asia, Web Search & Data Mining Group
			http://research.microsoft.com/wsm/
		+ Standford InfoLab, http://infolab.stanford.edu/
		+ UIUC Information Retrieval Group, http://sifaka.cs.uiuc.edu/ir/
		+ 北大天网组, http://sewm.pku.edu.cn/
		+ 北京大学计算语言学研究所, http://icl.pku.edu.cn/
		+ 复旦大学信息检索和自然语言处理组, 
			http://www.cs.fudan.edu.cn/mcwil/irnlp/
		+ 哈工大信息检索组, http://ir.hit.edu.cn/
		+ 清华大学智能技术与系统国家重点实验室
			http://www.csai.tsinghua.edu.cn/ 
		#+ 中科院大规模内容计算组, http://159.226.40.18/ (fail to visit)

	Researchers
		+ Andrew McCallum,
			http://www.cs.umass.edu/~mccallum/
		+ ChengXiang Zhai, developing Lemur
			http://www-faculty.cs.uiuc.edu/~czhai/
		+ Gerard Salton
			http://www.cs.cornell.edu/Info/Department/Annual95/Faculty/Salton.html
		+ Karen Sparck, developing IDF
			http://www.cl.cam.ac.uk/users/ksj/
		+ Keith van Rijsbergen
			http://www.dcs.gla.ac.uk/~keith/
		+ Jamie Callan, 
			http://www.cs.cmu.edu/~callan/
		+ Jon Kleinberg, developing HIT
			http://www.cs.cornell.edu/home/kleinber/
		+ Li Xiaoming, developing Tianwang & Infomall
		+ Nick Craswell, developing Terabyte Track
			http://research.microsoft.com/~nickcr
		+ Susan Dumais, developing LSI
			http://research.microsoft.com/~sdumais/
		+ Yiming Yang, developing text categorization
			http://www.cs.cmu.edu/~yiming/
		+ Stephen Robertson, 
			http://research.microsoft.com/users/robertson/
		+ Tefko Saracevic
			http://www.scils.rutgers.edu/~tefko/
		+ W. Bruce Croft
			http://ciir.cs.umass.edu/personnel/croft.html

	Research-related Resources
		+ http://www-faculty.cs.uiuc.edu/~czhai/research.html

	Software
		+ Apache Lucene: a full-featured text search engine library
			http://lucene.apache.org/java/docs/index.html
		+ Gate: a general architecture for text engineering
			http://gate.ac.uk/
		+ Lemur: A full-text search engine
			http://www.lemurproject.org/
		+ MG: A full-text search engine
			http://www.math.utah.edu/pub/mg/
		+ Porter Stemmer: English stemming algorithm
			http://www.tartarus.org/martin/PorterStemmer/
		+ Nutch: an open source web search engine
			http://sourceforge.net/projects/nutch/
		+ TSE: A Tiny Search Engine
			http://sewm.pku.edu.cn/src/TSE/

---------------------
References: 
[1] Information Retrieval Resources, http://www.sigir.org/resources.html
[2] http://ir.dcs.gla.ac.uk/resources.html
[3] http://www.cs.cmu.edu/~callan/Teaching/Resources.html
[4] Diekemar, Information Retrieval Links, Jan. 28, 1999. 
	http://web.syr.edu/~diekemar/ir.html
[5] 陈鸿标，网上研习信息检索，1999年11月. 
	http://159.226.40.18/freshman/resources/网上研习信息检索.doc
[6] 数据挖掘研究院, http://www.dmresearch.net/
[7] 语音自然语言在线, http://www.snlpinfo.com/index.php
[8] PKU SEWM Group, http://sewm.pku.edu.cn/
[9] http://www.cs.cmu.edu/~callan/Teaching/Resources.html
[10] http://icl.pku.edu.cn/member/lisujian/maincontent.htm
[11] http://www.cs.fudan.edu.cn/mcwil/irnlp/link.htm
[12] Robert Krovetz, A Guide to the Literature of Information Retrieval,
	http://159.226.40.18/freshman/resources/guide-to-ir-lit.ps
[13] ACM Digital Library, 
	http://portal.acm.org/portal.cfm
	http://acm.lib.tsinghua.edu.cn/acm/
[14] http://www.sigir.org/proceedings/Proc-Browse.html
[15] SIGIR,
	http://portal.acm.org/browse_dl.cfm?linked=1&part=series&idx=SERIES278&coll=portal&dl=ACM&CFID=72474811&CFTOKEN=69288563
[16] WWW, International World Wide Web Conference
	http://portal.acm.org/browse_dl.cfm?linked=1&part=series&idx=SERIES968&coll=portal&dl=ACM&CFID=72474811&CFTOKEN=69288563
[17] China Digital Journal Community, http://wanfang.calis.edu.cn/wf/szhqk/index.html



---------------------

More details are listed as follows
====================
CIIR 
(The Center for Intelligent Information Retrieval, 
美国Massachusetts大学的智能信息检索中心)
http://ciir.cs.umass.edu/

The Center for Intelligent Information Retrieval, a National Science 
Foundation-created S/IUCRC Center, is one of the leading information retrieval 
research labs in the world. The CIIR develops tools that provide effective 
and efficient access to large, heterogeneous, distributed, text and 
multimedia databases.

CIIR accomplishments include significant research advances in the areas of 
distributed information retrieval, information filtering, topic detection, 
multimedia indexing and retrieval, document image processing, terabyte 
collections, data mining, summarization, resource discovery, interfaces 
and visualization, and cross-lingual information retrieval.

The Center for Intelligent Information Retrieval continues to support the 
emerging information infrastructure, both through research and technology 
transfer. The goal of the CIIR is to develop tools that provide effective 
and efficient access to large, heterogeneous, distributed, text and 
multimedia databases. 

====================
Glasgow Information Retrieval Group
http://www.dcs.gla.ac.uk/ir/
由Keith van Rijsbergen率领的英国Glasgow大学信息检索研究小组。
这个小组理论和实践并重，旨在建造一个高效、新颖、成功的多媒体信息检索系统，
为终极用户服务。

The Information Retrieval Group led by Professor Keith van Rijsbergen has a 
vigorous programme of research, based on both theory and experiment, aimed at 
giving end-users novel, effective, and efficient access to the world of 
multi-media information. The group, part of the Department of Computing Science, 
University of Glasgow, has a strong research history in a wide area of 
information retrieval research from theoretical modelling of the retrieval 
process to advanced system building and to the user-oriented evaluation of 
information retrieval systems. The group's interests also include many areas 
of Web information retrieval such as link analysis, summarisation and the 
development of novel interaction techniques (e.g., ostension, implicit feedback 
and graphical visualisation). Our research preserves a strong emphasis on 
the evaluation of interactive IR systems, and the group maintains strong links 
with researchers in Human-Computer Interaction and Psychology.

------
Keith van Rijsbergen, http://www.dcs.gla.ac.uk/~keith/
英国格拉斯哥大学。概率IR的逻辑推理学派代表人，出版了著名的IR经典教材 
INFORMATION RETRIEVAL， 重点介绍用概率研究信息检的方法。

=====================
Cambridge NLIP Group 
(Natural Language and Information Processing Group)
http://www.cl.cam.ac.uk/Research/NL/

Research in NLIP has been done in the Computer Laboratory for nearly fifty years. 
The earliest work, by Roger Needham and Karen Sparck Jones, was on automatic 
thesaurus construction, in the context of document retrieval and machine translation. 
Subsequent research by Karen Sparck Jones during the 1960s and 70s focused on 
statistical approaches to retrieval and included innovative work on term 
weighting.  From the later 1970s research in language processing developed, 
with work on syntax, semantics and discourse processing,

------
Karen Sparck Jones, http://www.cl.cam.ac.uk/users/ksj/
Karen Sparck Jones has been one of the most influential figures in Computing 
since the 1950’s. Her work on Information Retrieval and Natural Language Processing 
has never been so central as it is are today, with its implications for 
search engine technology, the semantic web and even bioinformatics.

In 1972, Karen Sparck Jones published in the Journal of Documentation the paper 
which defined the term weighting scheme now known as inverse document frequency (IDF).

Karen Sparck Jones is emeritus Professor of Computers and Information at the 
Computer Laboratory, University of Cambridge. She has worked in automatic 
language and information processing research since the late fifties, 
and has many publications including several books, most recently `Evaluating 
Natural Language Processing Systems' with Julia Galliers, and `Readings in 
Information Retrieval', edited with Peter Willett. 

1988年度Salton奖得主。现代概率IR模型的另一创始人。在NLP、IR等领域都颇有建树，
而且做了大量的组织性工作。现在供职于英国剑桥大学计算机学院。

====================
LTI
CMU (Carnegie Mellon Universit) Language Technologies Institute,
http://www.lti.cs.cmu.edu/

The Language Technologies Institute (LTI) of the School of Computer Science at
Carnegie Mellon University conducts research and provides graduate education
in all aspects of language technology and information management. The LTI was
established in 1996, as an expansion of the Center for Machine Translation
(CMT).

The Center for Machine Translation (CMT) was a research branch of the School
of Computer Science devoted to basic and applied research in all aspects of
natural language processing, with a primary focus on machine translation,
speech processing, and information retrieval. Containing a unique mix of
academic and industrial researchers specializing in various aspects of
computer science, artificial intelligence, computational linguistics and
theoretical linguistics, the CMT provided a rich and diverse environment for
collaboration among faculty, staff, visiting scholars, and qualified students.

------
Lemur Toolkit
Lemur is a collection of search engine algorithms and information retrieval
applications used for IR research, development and education. Lemur provides a
rich query language that supports search against simple texts, structured
(XML) texts, and texts annotated with part-of-speech, named-entity, and other
annotations used in NLP and text-mining applications. Lemur's search engines
comfortably support collections ranging from a few gigabytes to a few
terabytes of text. The software is distributed under open-source license, and
is used widely in the IR research community.

====================
Standford InfoLab
http://infolab.stanford.edu/

The Stanford WebBase Project
http://dbpubs.stanford.edu:8091/~testbed/doc2/WebBase/

The Stanford WebBase project is investigating various issues in crawling,
storage, indexing, and querying of large collections of Web pages. The project
builds on the previous Google activity that was part of the DLI1 initiative.
The DLI2 WebBase project aims to build the necessary infrastructure to
facilitate the development and testing of new algorithms for clustering,
searching, mining, and classification of Web content.
====================
北大天网组, http://sewm.pku.edu.cn/

    北京大学网络实验室自1997年开始从事搜索引擎方面的研究与系统开发，
技术积累深厚，综合实力和学术影响在国内一直处于领先地位。我们研发的
“天网”搜索引擎系统是全国最有影响的出自校园的搜索引擎，从1997年10月
开始一直运行至今。“天网”在增量搜索技术、快速检索技术，海量信息存储
技术等方面都具有较强的优势，她的不断发展培育了一批批在海量网络文本
信息处理方面有实战经验的学生，受到中外IT企业的普遍欢迎。
    从2001年开始，本研究组在搜索引擎技术的基础上，展开了中国互联网
信息历史的收集与存档工作，形成了“中国互联网信息博物馆”，至今已
收藏20亿在不同时期出现过的中文网页，是目前全国规模最大的历史网页收藏
与回放系统。同时，我们还尝试了在其基础上进行多学科交叉的研究。

====================
中科院大规模内容计算组
http://159.226.40.18/

    信息检索小组主要针对文本信息的检索开展研究，多次参加TREC会议，
取得了很好的研究成果。小组开发的天罗检索系统在很多国家重要的信息部门
得到了广泛的应用，目前主要的研究方向包括WEB信息的获取，WEB信息检索等。
    信息分析小组的研究主要集中在大规模多源异构信息的分析与挖掘方面，
主要包括文本分类与聚类、信息过滤、个性化服务、自然语言问答和浅层
自然语言处理等。小组研制了一系列文本信息加工处理的实验平台，目前实验
平台可以通过主页中“成果演示”进行演示。值得一提的是小组开展的公开源码
计划，其中的高性能分词系统ICTCLAS得到了研究人员的广泛认同与使用。

====================
复旦大学信息检索和自然语言处理组, 
http://www.cs.fudan.edu.cn/mcwil/irnlp/ 	

大规模文本处理主要研究自然语言（特别是中文信息）的处理技术和方法，
包括二个方面内容：首先是基础性工作，主要是基础性的理论和算法, 包括
自动分词、未登录词识别、词性和概念标注、句法分析和语义分析等,也包括
语料库的搜集整理等；其次是中文信息处理的应用技术，包括自动索引、
文本检索、文本摘要、文本分类和文本过滤，特别是上述技术在网络环境下
的应用。这部分工作是文本方向的研究重点。

====================
HIT-IRLab, http://ir.hit.edu.cn/

    哈工大信息检索研究室 (HIT-IRLab) 成立于 2001 年 3月。研究方向
包括文本检索、问答系统、自动文摘、文本挖掘和语言分析等， 研究室以
语言分析为基础研究，以文本过滤为应用研究，以信息抽取为语言分析从
句子理解向 篇章理解的延伸，以句子检索为在语言分析和篇章理解的支持
下的智能化精准检索技术。 

====================
SIGIR（美国计算机学会信息检索特别兴趣小组）、
TREC（文本检索学术年会）
MUC（消息理解学术年会）
TIPSTER（美国国防部高级研究计划署的IR实践基地）

====================
北京大学计算语言学研究所
http://icl.pku.edu.cn/

    北京大学计算语言学研究所成立于1986年。致力于计算语言学理论、语言
信息处理的基础资源和应用技术三方面的研究。
    围绕计算语言学和自然语言处理，包括如下三个主要的方向：首先基础资源
的研究与建设：计算词典学与机器词典，综合型语言知识库，语料库语言学与
语料库加工技术，术语学、术语自动提取、术语标准化研究等。其次是基础理论、
NLP的模型和方法：计算语言学基础，自然语言处理核心技术，现代汉语语法，
汉语的词/句法/语义分析，NLP统计模型，语言处理的信息论方法等。另外是
应用技术：机器翻译的方法、技术与系统实现，信息检索与提取，自然语言
信息处理系统的评价方法和技术，受限汉语及其辅助写作系统，中国古诗词计算机
辅助研究等。

====================
清华大学智能技术与系统国家重点实验室
http://www.csai.tsinghua.edu.cn/ 

    智能技术与系统国家重点实验室依托于清华大学。实验室于1990年2月
对外开放运行。主要从事人工智能基本原理、基本方法的基础与应用基础研究，
包括智能信息处理、机器学习、智能控制，以及神经网络理论等，还从事与
人工智能有关的应用技术与系统集成技术的研究，主要有智能机器人、声音、
图形、图像、文字及语言处理等。

================
Susan Dumais, 
http://research.microsoft.com/~sdumais/

I am interested in algorithms and interfaces for improved information
retrieval, as well as general issues in and human-computer interaction. I
joined Microsoft Research in July 1997. I work on a wide variety of
information access and management issues, including: personal information
management, web search, question answering, information retrieval, text
categorization, collaborative filtering, interfaces for improved search and
navigation, and user/task modeling.

Prior to coming to Microsoft, I worked on a statistical method for
concept-based retrieval known as Latent Semantic Indexing. You can find
pointers to this work on the Bellcore (now Telcordia) LSI page. 

===============
UIUC Information Retrieval Group
http://sifaka.cs.uiuc.edu/ir/

The Information Retrieval (IR) group is part of the Database and Information
Systems (DAIS) Lab  of the Computer Science Department at University of
Illinois at Urbana-Champaign. We work on a wide spectrum of problems in the
general area of text information management, including  retrieval,
organization, filtering , and mining of textual information, aiming at
developing advanced text information management techniques and systems that
help people make better use of text information.

------
ChengXiang Zhai, 
http://www-faculty.cs.uiuc.edu/~czhai/

Research Interests: Information Retrieval, Text Mining, Natural Language
Processing, Bioinformatics

University of Illinois at Urbana-Champaign, is recognized for
his work on user-centered, adaptive intelligent information access. His
techniques expect to improve search-engine performance, support better
information organization and enable understanding of large volumes of
information. Zhai's work in information retrieval is expected to enhance
curricula and provide new educational tools for the growing information
technology workforce.

===============
Stephen Robertson, 
http://research.microsoft.com/users/robertson/

Stephen Robertson joined Microsoft Research Cambridge in April 1998.

In 1998, he was awarded the Tony Kent STRIX award by the Institute of
Information Scientists. In 2000, he was awarded the Salton Award by ACM SIGIR.
He is a Fellow of Girton College, Cambridge.

At Microsoft, he runs a group called Information Retrieval and Analysis, which
is concerned with core search processes such as term weighting, document
scoring and ranking algorithms, and combination of evidence from different
sources. These are studied theoretically through the use of formal models,
mainly statistical, and statistical methods including machine learning
methods, and experimentally, through activities such as the Text Retrieval
Conference (TREC) and with internally generated evaluation sets. The group
(with its Keenbow evaluation environment) has had some excellent results at
TREC. The group works closely with product groups to transfer ideas and
techniques.

His main research interests are in the design and evaluation of retrieval
systems. He is the author, jointly with Karen Sparck Jones, of a probabilistic
theory of information retrieval, which has been moderately influential. A
further development of that model, with Stephen Walker, led to the term
weighting and document ranking function known as Okapi BM25, which is used in
many experimental text retrieval systems.

Prior to joining Microsoft, he was at City University London, where he retains
a part-time position as Professor of Information Systems in the Department of
Information Science (homepage). He was Head of Department for eight years,
during which time it achieved the highest possible rating in two successive
research assessment exercises. He also started the Centre for Interactive
Systems Research, the main research vehicle of which is the Okapi text
retrieval system, which has also done well at TREC.

Before joining City, he was a research fellow at University College London,
where he took his PhD in the School of Library Archive and Information
Studies. Before that he was in the research department at Aslib. He has an MSc
in Information Science from City and a first degree in mathematics from
Cambridge. 

===================
Nick Craswell
http://research.microsoft.com/~nickcr

I am an associate researcher at Microsoft Research Cambridge, in the
Information Retrieval and Analysis Group.

Research Overview

I am interested in Web search evaluation, mostly on enterprise-scale webs but
also the World Wide Web. I built the VLC, VLC2, WT2g and .GOV test
collections, which have been made available to research groups around the
world. David Hawking and I coordinated the TREC Web Track experiments. I am
currently involved in the TREC Terabyte Track and Enterprise Track. Some
publications: Book chapter preprint (pdf), IR'01 (citeseer) and CSIRO'01
(pdf).

I also work on effective Web search, which means making use of information in
pages, link structure and URL structure to generate more useful Web search
results. Some papers: SIGIR'05 (pdf), SIGIR'01 (pdf), TOIS'03 (pdf) (copying
is by permission of ACM, Inc.) and ADCS'03 (pdf).

My PhD was in distributed information retrieval (thesis pdf) which means
building a system on top of multiple engines/databases that already exist. My
recent work in the area has considered whether (or when) DIR is really
practical. Some papers: ADC'99 (ps), DL'00 (pdf), ADC'03 (pdf) and ADC'04
(pdf). 

===============
Web Search & Data Mining Group of MSR Asia
http://research.microsoft.com/wsm/

The goal of the Web Search & Data Mining Group of MSR Asia is to drive the
next generation of Web search by leveraging data mining, machine learning, and
knowledge discovery techniques for information analysis, organization,
retrieval, and visualization. In addition, in contrast with current Web search
methods, which essentially do document-level ranking and retrieval, the Web
Search & Data Mining Group has created search at the object level to bring
increased knowledge and intelligence to users.

A Glimpse at Several Core Innovations:

Large-scale Experimental Web Search Platform

The Web Search & Data Mining Group is creating a large scale search platform
to efficiently store, parse, index and search billions of Web pages and other
types of documents. The search platform is flexible enough to allow for
testing of various state-of-the-art search techniques that have been created
at the lab using new technologies.

Structuralizing the Web

The biggest challenge facing both users and search engines over the next
several decades is the continued unstructured growth of the Internet. As such,
search functions that can effectively and efficiently dig out
machine-understandable information and knowledge layers from unorganized and
unstructured Web data will be the key to supporting relevant search results.
To meet this challenge, the group is exploring technologies, namely Web
information extraction, deep Web mining, and Web structure mining that can
automatically classify structures and extract objects from the Web. The
information and knowledge gathered using these new techniques greatly improves
the performance of current Web search and even facilitates the creation of
more sophisticated next generation search technologies.

Vertical Search

Today's conventional search engines can be described as page-level search
engines whose main function is to rank web pages according to their relevance
to a given query. Driving the future of the search industry are functions that
delve deeper into vertical domains to provide knowledge and intelligence to
query results. At MSR Asia, the Web Search & Data Mining Group is addressing
the greatest challenges faced by vertical search including large scale web
classification, object-level information extraction, object identification and
integration, and object relationship mining and ranking. The results of these
efforts are leading to more advanced search engines that deliver intelligence
and insight to search results.

Mobile Search

The explosive growth of new computing devices such as handheld computers,
Windows Mobile-based PocketPCs, and SmartPhones is driving demand for greater
and more efficient information access. These devices, which leverage the power
of the Web and allow greater access to information than ever before, are still
not capable of performing at the level of a desktop PC. At MSR Asia, the Web
Search & Data Mining Group is inventing new technologies to improve the mobile
search and browsing experience and deliver the capabilities of a PC to users
of these new devices. Project initiatives include developing innovative
presentation schemes and user interfaces to facilitate search and browsing
tasks on mobile devices and developing context aware search technologies to
address the special information needs of mobile users.

Multimedia Search

The Web Search & Data Mining Group is conducting research into new
technologies that index multimedia content such as images, videos, and audio.
Through content analysis and advanced visualization techniques, the group is
transforming today's conventional text based search engines to include
multimedia content thus delivering more intelligent search results to users.
For example, the group recently developed a new multimedia news reader which
mines large archival news databases presenting text, map information, images,
and background music within a unique user interface providing readers with a
more efficient news search engine and a more enjoyable reading experience.

------
Wei-Ying Ma
http://research.microsoft.com/users/wyma/

Senior Researcher, Research Manager, Microsoft Research Asia

Dr. Wei-Ying Ma received the B.S. degree in electrical engineering from the
National Tsing Hua University in Taiwan in 1990, and the M.S. and Ph.D.
degrees in electrical and computer engineering from the University of
California at Santa Barbara in 1994 and 1997, respectively. From 1994 to 1997
he was engaged in the Alexandria Digital Library (ADL) project in UCSB while
completing his Ph.D. He developed a web-based image retrieval system called
Netra which has been frequently cited by other researchers and is regarded as
one of the most representative image retrieval systems. From 1997 to 2001, he
was with HP Labs where he worked in the field of multimedia adaptation and
distributed media services infrastructure. He joined Microsoft Research Asia
in 2001. Since then, he has been leading a research group to conduct research
in the areas of information retrieval, web search, data mining, mobile
browsing, and multimedia management. He currently serves as an Editor for the
ACM/Springer Multimedia Systems Journal and Associate Editor for ACM
Transactions on Information System (TOIS). He has served on the organizing and
program committees of many international conferences including ACM Multimedia,
ACM SIGIR, ACM CIKM, WWW, ICME, CVPR, SPIE Multimedia Storage and Archiving
Systems, SPIE Multimedia Communication and Networking, etc. He is also the
general co-chair of International Multimedia Modeling (MMM) Conference 2005
and International Conference on Image and Video Retrieval (CIVR) 2005. He has
published 5 book chapters and over 100 international journal and conference
papers.

====================
Google Labs
http://labs.google.com/

Google Labs is a playground for Google engineers and adventurous Google users.
Google staffers with wild and crazy ideas post their prototypes on Google Labs
and solicit feedback on how the technology could be used or improved. None of
these experiments are guaranteed to make it onto Google.com, as this is really
the first phase in the development process. Google users with a desire to jump
over the cutting edge are invited to check out any or all of the posted
prototypes and send their comments directly to the Googlers who developed
them. Please, remember to wear your safety goggles while using this site.

Labs.google.com, Google's technology playground.
Google labs showcases a few of our favorite ideas that aren't quite ready for
prime time. Your feedback can help us improve them. Please play with these
prototypes and send your comments directly to the Googlers who developed them. 

Want to learn more about Google technology? Here are some papers.
http://labs.google.com/papers/index.html

Passionate about these topics? You should work at Google.
algorithms, artificial intelligence, compiler optimization,
computer architecture, computer graphics,
data compression, data mining, file system design,
genetic algorithms, information retrieval,
machine learning, natural language processing, operating systems,
profiling, robotics, 
text processing, user interface design,
web information retrieval, and more! 

http://www.google.com/press/podium.html
Google Press Center: The Google Podium
 Here you'll find a selection of public presentations made by Google
executives. From time to time, we will continue to add transcripts, audio or
video clips and links to presentations hosted elsewhere.

====================
Jon Kleinberg
http://www.cs.cornell.edu/home/kleinber/

Professor of Computer Science, Cornell University

My research is concerned with algorithms that exploit the combinatorial
structure of networks and information. My recent work has included
* link analysis and modeling of the World Wide Web and related information networks;
* discrete optimization and network algorithms; and
* algorithmic approaches to clustering, indexing, and data mining. 
====================

    一、概论

    作为Hadoop程序员，他要做的事情就是：
    1、定义Mapper，处理输入的Key-Value对，输出中间结果。
    2、定义Reducer，可选，对中间结果进行规约，输出最终结果。
    3、定义InputFormat 和OutputFormat，可选，InputFormat将每行输入文件的内容转换为Java类供Mapper函数使用，不定义时默认为String。
    4、定义main函数，在里面定义一个Job并运行它。
   

    然后的事情就交给系统了。
    1.基本概念：Hadoop的HDFS实现了google的GFS文件系统，NameNode作为文件系统的负责调度运行在master，DataNode运行在每个机器上。同时Hadoop实现了Google的MapReduce，JobTracker作为MapReduce的总调度运行在master，TaskTracker则运行在每个机器上执行Task。

    2.main()函数，创建JobConf，定义Mapper，Reducer，Input/OutputFormat 和输入输出文件目录，最后把Job提交給JobTracker，等待Job结束。

    3.JobTracker，创建一个InputFormat的实例，调用它的getSplits()方法，把输入目录的文件拆分成FileSplist作为Mapper task 的输入，生成Mapper task加入Queue。

    4.TaskTracker 向 JobTracker索求下一个Map/Reduce。
      
     Mapper Task先从InputFormat创建RecordReader，循环读入FileSplits的内容生成Key与Value，传给Mapper函数，处理完后中间结果写成SequenceFile.
     Reducer Task 从运行Mapper的TaskTracker的Jetty上使用http协议获取所需的中间内容（33%），Sort/Merge后（66%），执行Reducer函数，最后按照OutputFormat写入结果目录。

      TaskTracker 每10秒向JobTracker报告一次运行情况，每完成一个Task10秒后，就会向JobTracker索求下一个Task。

      Nutch项目的全部数据处理都构建在Hadoop之上，详见Scalable Computing with Hadoop。

    二、程序员编写的代码

    我们做一个简单的分布式的Grep，简单对输入文件进行逐行的正则匹配，如果符合就将该行打印到输出文件。因为是简单的全部输出，所以我们只要写Mapper函数，不用写Reducer函数，也不用定义Input/Output Format。

        RegMapper类的configure()函数接受由main函数传入的查找字符串，map() 函数进行正则匹配，key是行数，value是文件行的内容，符合的文件行放入中间结果。
        main()函数定义由命令行参数传入的输入输出目录和匹配字符串，Mapper函数为RegMapper类，Reduce函数是什么都不做，直接把中间结果输出到最终结果的的IdentityReducer类，运行Job。

        整个代码非常简单，丝毫没有分布式编程的任何细节。

       三.运行Hadoop程序

        Hadoop这方面的文档写得不全面，综合参考GettingStartedWithHadoop 与Nutch Hadoop Tutorial 两篇后，再碰了很多钉子才终于完整的跑起来了，记录如下：      

3.1 local运行模式

       完全不进行任何分布式计算，不动用任何namenode,datanode的做法，适合一开始做调试代码。
       解压hadoop，其中conf目录是配置目录，hadoop的配置文件在hadoop-default.xml，如果要修改配置，不是直接修改该文件，而是修改hadoop-site.xml，将该属性在hadoop-site.xml里重新赋值。
       hadoop-default.xml的默认配置已经是local运行，不用任何修改，配置目录里唯一必须修改的是hadoop-env.sh 里JAVA_HOME的位置。

       将编译好的HadoopGrep与RegMapper.class 放入hadoop/build/classes/demo/hadoop/目录 找一个比较大的log文件放入一个目录，然后运行

     查看输出目录的结果，查看hadoop/logs/里的运行日志。  
     在重新运行前，先删掉输出目录。
 

3.2 单机集群运行模式

       现在来搞一下只有单机的集群.假设以完成3.1中的设置，本机名为hadoopserver
       第1步.    然后修改hadoop-site.xml ，加入如下内容：

    从此就将运行从local文件系统转向了hadoop的hdfs系统，mapreduce的jobtracker也从local的进程内操作变成了分布式的任务系统，9000，9001两个端口号是随便选择的两个空余端口号。
 
  另外，如果你的/tmp目录不够大，可能还要修改hadoop.tmp.dir属性。

  第2步. 增加ssh不输入密码即可登陆。

    因为Hadoop需要不用输入密码的ssh来进行调度，在不su的状态下，在自己的home目录运行ssh-keygen -t rsa ,然后一路回车生成密钥，再进入.ssh目录,cp id_rsa.pub authorized_keys
    详细可以man 一下ssh, 此时执行ssh hadoopserver，不需要输入任何密码就能进入了。

  3.格式化namenode，执行
  bin/hadoop namenode -format

  4.启动Hadoop
     执行hadoop/bin/start-all.sh, 在本机启动namenode,datanode,jobtracker,tasktracker
 
  5.现在将待查找的log文件放入hdfs,。
     执行hadoop/bin/hadoop dfs 可以看到它所支持的文件操作指令。
     执行hadoop/bin/hadoop dfs put log文件所在目录 in ，则log文件目录已放入hdfs的/user/user-name/in 目录中

  6.现在来执行Grep操作
      hadoop/bin/hadoop demo.hadoop.HadoopGrep in out
      查看hadoop/logs/里的运行日志，重新执行前。运行hadoop/bin/hadoop dfs rmr out 删除out目录。

  7.运行hadoop/bin/stop-all.sh 结束

  3.3 集群运行模式
  假设已执行完3.2的配置，假设第2台机器名是hadoopserver2
  1.创建与hadoopserver同样的执行用户，将hadoop解压到相同的目录。

  2.同样的修改haoop-env.sh中的JAVA_HOME 及修改与3.2同样的hadoop-site.xml

  3. 将hadoopserver中的/home/username/.ssh/authorized_keys 复制到hadoopserver2,保证hadoopserver可以无需密码登陆hadoopserver2
     scp /home/username/.ssh/authorized_keys  username@hadoopserver2:/home/username/.ssh/authorized_keys
 
  4.修改hadoop-server的hadoop/conf/slaves文件, 增加集群的节点，将localhost改为
    hadoop-server
    hadoop-server2

  5.在hadoop-server执行hadoop/bin/start-all.sh
   将会在hadoop-server启动namenode,datanode,jobtracker,tasktracker
   在hadoop-server2启动datanode 和tasktracker
 
  6.现在来执行Grep操作
     hadoop/bin/hadoop demo.hadoop.HadoopGrep in out
    重新执行前,运行hadoop/bin/hadoop dfs rmr out 删除out目录

  7.运行hadoop/bin/stop-all.sh 结束。
   

四、效率

    经测试，Hadoop并不是万用灵丹，很取决于文件的大小和数量，处理的复杂度以及群集机器的数量，相连的带宽，当以上四者并不大时，hadoop优势并不明显。
    比如，不用hadoop用java写的简单grep函数处理100M的log文件只要4秒，用了hadoop local的方式运行是14秒，用了hadoop单机集群的方式是30秒，用双机集群10M网口的话更慢，慢到不好意思说出来的地步。

¹（西安电子科技大学电子工程学院　西安 710071）

²（西安电子科技大学经济管理学院　西安 710071）

【摘要】在对Lucene的索引机制进行分析的基础上，对其在索引的构建策略，索引数据结构，排序算法，中文分词等方面进行优化改进，以期可以适应于大规模信息的索引处理。

【关键词】Lucene 倒排索引 搜索引擎 全文检索

1 引言

Lucene是基于java的高性能的全文检索工具包，其提供了一组功能强大的API，在此基础上程序员可以快速方便的构建全文检索系统。此外，Lucene还可以嵌入到其它的应用系统中，根据具体的应用环境，实现全文检索功能。实践证明Lucene可以胜任一些大型的应用^[1,2,4]，如Apache组织网站的检索系统是构建在Lucene之上的，IBM的Eclipse，WebSphere也采用Lucene全文索引系统，而且Lucene提供了丰富的接口，应用程序员可以根据需要进行扩展，加入适合实际需求的新功能。

文中的第二部分探讨了Lucene索引机制，针对索引的数据组织结构和构建流程等进行了分析；第三部分给出了一下索引的优化策略，分别就内存缓存，索引的压缩编码，中文的处理技术和排序算法给出了相关的优化方案；最后是总结和展望以及对下一步的研究。

2 Lucene索引机制

2.1 Lucene索引基本原理

       Lucene是一组全文检索API，在Lucene索引中，其基本的概念为词（Term）、域（Field）、文档（Document）、段（Segment）、索引（Index），它们之间的关系如图1所示^[3]。

图1  Lucene索引的组织结构

       Lucene索引（Index）由若干的段（Segment）组成，每个段都是一个子索引，段中包括若干的文档（Document），文档代表检索结果的一个实体，比如网页等，文档又由域（Field）组成，域是词（Term）的集合，词是索引的最小单位，就是文档中的单词，而域就是就有一定属性关系的词的集合，比如URL，作者，标题等都可以设置为域。

Lucene采用倒排索引的全文索引策略，其索引的数据结构为（Term，（DocId，（frq，pos））…）。

在图2中，给出了Lucene索引工具包的系统架构图，由图可以看出，Lucene由索引核心，对外接口，基础结构类模块组成，主要分为七个包，分别负责索引的构建，索引的存储，文档的分析，查询接口，文档的解析，一些实用类集合，索引的数据结构等^[5]。

图2 Lucene系统架构

基于Bloom Filter的大规模网页去重策略研究

【摘要】Bloom Filter是一种空间高效的集合查找和表示算法。针对大规模信息采集，运用Bloom Filter及其改进算法，在误差允许的条件下，通过URL散列运算可以有效地对同源网页进行去重。实验证明通过对其参数进行合理的调整，可以达到满意的结果。

【关键词】布隆过滤器，散列函数，URL，网页去重

【分类号】 TP391.3

The Research of large-scale URL Filter Based on Bloom Filter

【Abstract】Bloom Filter is a space-saving set search and represent algorithm for large-scale information collection. The Bloom Filter and its improvement algorithm, on the condition of error allowing, can be used to the homology URL page filter through URL Hashing. Experiment has proved that it can achieve satisfactory results through reasonable adjustments of its parameter.

【Keywords】 Bloom Filter，Hash Function，URL，URL  Filter

1引言

    Web信息的采集，通常是利用网络爬虫等工具去遍历WWW，它把WWW看作一个以网页为节点，网页间链接为边的超大规模有向图，然后利用图的遍历算法对WWW进行遍历。在网络遍历的过程中，需要判断待采集的页面是否已经采集过了，这就需要把已经采集的网页地址记录下来，组成已采集网页地址集合（记为：visited-set），当新的采集开始之前，首先判断其地址是否在visited-set中，如在其中，表示网页已经采集，否则采集网页，把网页地址放visited-set中，从而避免网页的重复采集，浪费资源。为了实现集合中数据的快速查找，需要把URL映射为集合中的地址，这就需要设计一种高效且冲突率低的散列算法；同时由于WWW上网页数据的巨大（一份报告指出，截止2005年1月份其网页数量至少达到115亿^[1]），普通的Hash算法已经不能满足空间的要求，所以又需要一种节约空间的算法。

     本文运用Bloom Filter算法设计了一种节省空间的大规模数据表示和查找算法，以应对海量信息采集的需求。文章的组织结构为：第2部分对Bloom Filter算法进行简要的介绍，包括算法思想、误判率计算以及哈希函数选取策略；第3部分给出了Bloom Filter及其改进算法在大规模网页去重策略中的应用；第4部分通过实验对Bloom Filter算法在网页去重中的应用给出了可行性分析；第5部分进行的总结以及进一步的研究工作。

2 Bloom Filter算法

2.1 Bloom Filter算法简介

Bloom Filter是由巴顿布隆于1970年提出的^[2]，它实现的基础上是一个很长的二进制位向量和一系列随机散列函数。Bloom Filter是一种基于散列的查找算法，用于查找一个元素是否在集合中，和散列表相比，它的优点是节约空间，可以对海量数据集进行表示和查找操作。由于散列函数的随机性，可能使得某个元素不属于集合而被判定属于集合，在此，称其为误判，其大小为误判率P_err。

Bloom Filter算法的基本思想为：

①设数据集合A={a­­₁,a₂,…,a_n}，含有n个元素，为待操作的集合；

②Bloom Filter用一个长度为m的位向量V来表示集合中的元素，位向量初始化全为0；

③ k个具有均匀分布特性的散列函数h1 h2,,…,hk ，值域均为{ 1 , 2 ,⋯, m} ；

④ 对于元素的加入操作首先通过K个散列函数产生K个随机数h1,h2,…,hk，使位串V的相应h1,h2,…,hk位均置为1；同理，元素的查找为判定相应位是否全为1。

2.2 Bloom Filter误差分析

Bloom Filter算法在进行集合的元素表示时，由于通过k个散列函数使位向量相应位置1，经过多次集合元素增加操作后，使得位向量若干位被重复置为1。当新的元素加入集合之前，首先通过集合查找运算判定元素是否在集合中，同样，通过k个散列函数的映射到相应的位，如果相应位都为1，则判定元素在集合中，否则判定元素不在集合中。对于已经映射到集合中的元素，显然通过集合查找运算一定可以判定其中集合中，但对于尚未映射到集合中的元素，可能存在误判，即不在集合中的元素误判为在集合中。

下面我们对误判率进行分析：

在Bloom Filter表示方法中，某一位被置为1个概率为 ，为0的概率为1- ,散列函数共执行了kn次，所以在运算结束时，某位仍为0的概率为：

P₀=（1- ）^kn e^-kn/m                  (式1)

所以误判的概率为：

P_err=(1- P₀)^k=(1-(（1- ）^kn))^k (1- e^-kn/m)^k    (式2)

当给定k和n时，由式（2）可知随着m的增大P_err减小；对于给定的n和m，下面求 P_err的最小值，对式（2）两端求导，令 =0，可求得当k= ln2 0.7 时， P_{err_min} 0.6185^m/n为最小的误判率。

2.3 散列函数考虑

散列(Hashing)是信息表示和查找所用的一项基本技术^[3]，其优化性能在很大程度上取决于输入键的结构，本文所研究散列的键是用于访问网页的URL。在运用Bloom Filter算法进行集合的表示和元素的查找过程中，需要对待查找（或表示）的数据进行k次散列操作，所以散列函数的选取成为了影响系统性能的关键因素。针对web信息的采集，文献[4]评价比较了五种Hash函数，并对它们在URL映射的性能进行了比较分析，结果显示Strhash和Tianlhash的性能较佳。文献[5] 给出了两种针对URL散列性能很好的函数，并通过2000万URL的实验进行了评价，结果表明,HfIp 都是很可靠的,可以首选使用，并在北大天网搜索引擎系统中得到工程性的检验。