Modernizing Scala The Journey to Java 8 Jason Zaugg
1
2
Overview Scala 2.12 Trait Encoding invokedynamic Lambda Encoding
3
Scala 2.12
4.1
Scala 2.12 Drop Java 6/7 support Exploit new features in JVM Improve Java interop New code generator and -optimise implementation Non Goals: new language features, major library changes
4.2
Java Support/Target Matrix Scala 2.11
Java 6
Java 8
ST
S
Scala 2.12.0
ST
Scala 2.12.N
ST
Java 9
S
4.3
Code Generator previously: AST -> IR -> optimized IR -> bytecode now: AST -> bytecode -> optimized bytecode AOT optimization (if enabled): standard local optimizations: jump chain, dce, nullness, pushpop, redundant store etc. Scala aware box/unbox elimination (tuples, options) inlining (w/ Scala aware CHA)
4.4
Interop Scala lambdas now fulfil any functional interface (FI) Scala traits with concrete methods can be implemented by Java classes hence, FunctionN are now FIs
4.5
Consume Java Functional APIs Use functional APIs from Java libraries, e.g. j.u.stream scala> import java.util.stream._ import java.util.stream._ scala> Stream.of("a", "b", "c").map[String](_.toUpperCase).findFirst.get res0: String = A
4.6
Consume Java Functional APIs Use site variance in Stream API doesn't play nicely with inference. scala> Stream.of("a", "b", "c").map(_.toUpperCase).findFirst.get res8: ?0 = A
Retrofit declaration site variance? scala> :power scala> val List(i, o) = symbolOf[Function[_, _]].typeParams i: $r.intp.global.Symbol = type T o: $r.intp.global.Symbol = type R scala> i.setFlag(Flags.CONTRAVARIANT); o.setFlag(Flags.COVARIANT) res6: o.type = type R scala> Stream.of("a", "b", "c").map(_.toUpperCase).findFirst.get res8: String = A
4.7
Expose Scala functional APIs Java lambdas can fulfil scala.FunctionN jshell> new scala.Some("42").map(x -> x + "!!!").get() $1 ==> "42!!!"
4.8
Smaller JARs lambda lovers profit the most YMMV
4.9
Encoding Traits without the bloat
5.1
Traits Cheat Sheet classes may extend one class + many traits traits may contain anything that a class does: initialization, fields, method bodies no constructor params (yet)
5.2
2.11 traits Trait: interface T including abstract accessors for fields class T$class with method bodies Subclass: implement interface methods by forwarding to impl class fields + concrete accessors
5.3
2.11 traits trait T { def foo = 42; var v = 1 } class C extends T
Compile to: public interface T { int foo(); int v(); void v_$eq(int); } class T$class { static int foo(T $this) { return 42; } static void $init$(T) { T.v_eq(1); } } class C implements T { C() { T.$init$(this); } private int v; int v() { return v; } void v_eq(int v) { this.v = v } int foo() { return T$class.foo(this); } }
5.4
2.12 traits (naive) field encoding unchanged leave trait method bodies in the interface trait constructor as a static method omit forwarders in classes
5.5
2.12 traits (naive) trait T { def foo = 42 } class C extends T
Compile to: interface T { default int foo() { return 42; } } class C implements T
5.6
Challenge: runtime dispatch class C { def foo = "C" } trait T extends C { override def foo = "T" } class D extends T new D().foo // prints?
5.7
Challenge: runtime dispatch JVM and Scala disagree on dispatch here We need to add a "trait forwarder" to D class D extends T { /*synthetic*/ def foo = super[T].foo }
only add in case of disagreement
5.8
Trait forwarders trait forwarders need to name arbitrary ancestor trait JVM only allows named parents Add redundant parents?
5.9
the parent of my parent is my parent Add all ancestors to parent interface list What could possibly go wrong?!
5 . 10
the parent of my parent is my parent Startup time ballooned HierarchyVisitor in vm/classfile/defaultMethods.cpp uncooperative
5 . 11
Challenge: super calls what about explicit super calls? super calls in classes follow Java's restrictions but in traits, super calls can target any ancestor
5 . 12
Challenge: super calls Simple case: trait T { def foo = 42 } trait U extends T { override def foo = -super[T].foo } class C extends U
Compile to: interface T { default int foo() { return 42; } } interface U extends T { default int foo() { return super.foo(); } } class C implements U
5 . 13
super dispatch Scala super requires static dispatch invokespecial dispatches virtually Latent bug in Scala 2.11 SD-143
5 . 14
class A { def m = 1 } class B extends A { override def m = 2 } trait T extends A class C extends B with T { override def m = super[T].m }
5 . 15
class A { def m = 1 } class B extends A { override def m = 2 } trait T extends A class C extends B with T { override def m = invokespecial T.m }
5 . 16
class A { def m = 1 } class B extends A { override def m = 2 } trait T extends A class C extends B with T { override def m = invokeplease A.m // }
5 . 17
class A { def m = 1 } class B extends A { override def m = 2 } trait T extends A class C extends B with T { override def m = invokeactually B.m // }
5 . 18
super-dilemma Indy to the rescue? bootstrap could unreflectSpecial would require elevated permissions Or, add static accessors for all trait methods we can then use invokestatic as before can't target Java defaults or class methods introduce language restriction for such cases 5 . 19
Cleaner stacks 2.11: scala> trait T { def foo = throw null }; new T {}.foo java.lang.NullPointerException at T$class.foo(:11) at $anon$1.foo(:13) ... 32 elided
2.12: scala> trait T { def foo = throw null }; new T {}.foo java.lang.NullPointerException at T.foo(:11) ... 30 elided
5 . 20
Traits Recap we've shed trait impl classes and most trait forwarders shallower, less crufty call stacks non-private methods still require static accessors reconsider the indy-super solution down the track
5 . 21
arriving (fashionably late) at
the indy party
6.1
@PolymorphicSignature scala> import java.lang.invoke._, MethodType._ scala> val lookup = MethodHandles.lookup(); import lookup._ scala> findVirtual(classOf[Object], "hashCode", methodType(classOf[Int])) res0: java.lang.invoke.MethodHandle = MethodHandle(Object)int scala> val hc: Int = res0.invoke(new Object) hc: Int = 1251285265
support added in 2.11 series
6.2
Symbol Literals 'foo // shorthand for scala.Symbol("foo") def x = f('blah) x eq q
We used to cache in synthetic static field private statics not supported in interfaces publicise? indify!
6.3
Structural calls (c: {def close(): Unit}) => c.close()
Compiled to reflective calls + reflection cache again, indify the cache into the callsite future work: dynalink first, need to teach it about Int.+ etc
6.4
indy liberation scalac backend emits indy for new AST shape not special cased to use cases above macros: transform application => arbitrary AST (undocumented, experimental) library level support for indy
6.5
Scala macro primer Experimental feature of Scala 2.{10,11,12} Client code looks, feels, typechecks like an method call Macro author writes a pair of methods: a declaration with the type signature an impl that transforms the AST stdlib macros: string formatting, quasiquotes quasiquote macro shorthands AST de-/con-struction
6.6
Example: link time pattern compilation
6.7
patterns: Client Code // pattern is compiled and cached when call site is linked compilePattern("foo.bar").matcher("foo!bar").matches // bonus static error: Unclosed character class compilePattern("abc[..")
6.8
patterns: Macro class impl(val c: Context) extends IndySupport { import ... val bootstrapSym = typeOf[test.Bootstrap].companion .member(TermName("bootstrap")) val invokedType = NullaryMethodType(typeOf[Pattern]) def compilePattern(s: Tree): Tree = s match { case l@Literal(Constant(s: String)) => checkPattern(l.pos, s) Indy(bootstrapSym, List(l), invokedType) case _ => q"_root_.java.util.regex.Pattern.compile($s)" } } def compilePattern(s: String): Pattern = macro impl.compilePattern
6.9
patterns: Bootstrap CallSite bootstrap(Lookup lookup, String invokedName, MethodType invokedType, String value) { return new ConstantCallSite( MethodHandles.constant(Pattern.class, Pattern.compile(value))); }
6 . 10
indy liberation See John Rose's ideas: mlvm-dev/2016-July/006661.html Scala's macro system can help explore these ideas
6 . 11
Example: Java 8/9 shims Could we use this to help people cross target Java 8/9? Link-time choice of Unsafe vs VarHandle
6 . 12
shims: Client Code class Foo { var blah = 42 } import Macro._ val f = new Foo() compareAndSetInt(classOf[Foo], "blah", f, -1, 0) assert(!compareAndSetInt(classOf[Foo], "blah", f, -1, 0)) assert(compareAndSetInt(classOf[Foo], "blah", f, 42, 0)) assert(f.blah == 0)
6 . 13
autostaging? def foo(x: String) = { partiallyEvaluate { List("1", "2", "3").contains(x)) } }
Idea: macro partiallyEvaluate could mechanically turn this into: def foo(x: String) = { invokedynamic[LinkTimeInterpreter, List("1", "2", "3")].contains(x) }
6 . 14
autostaging? Lots of related research in Scala world, although "Lightweight Modular Staging", Rompf, Odersky "Yin-Yang: Concealing the Deep Embedding of DSLs", Jovanovic et al integration with link time eval deserves more research
6 . 15
indyλ
7.1
Lambdas: Before and after Essentially the same as Java 7 => 8
7.2
Challenge: LMF assumptions Can't define toString Unboxing of nulls in Scala is 0, not NPE Scala Value Class boxing void return type needs to be boxed
7.3
Challenge: LMF assumptions scala> (() => ()).toString res1: String = [function0] scala> ((x: Int) => x).asInstanceOf[Any => Int].apply(null) res2: Int = 0 scala> ((() => ()) : (() => Any)).apply().getClass res3: Class[_] = class scala.runtime.BoxedUnit
7.4
Challenge: LMF assumptions Solution: live with the new toString selectively use adapter methods at the lambda target
7.5
Challenge: Scala Specialization Specialized traits leave the wrong method abstract Should change specialization to fix this but it is a complex beast, and we've started out with intermediate FIs to flip the methods.
7.6
Challenge: Serialization javac treats serializable func interfaces as the exception scala lambdas are typically serializable javac-style $deserializeLambda$ would grow large hard limit on number of lambdas (JDK-8008413) use a generic lambda deserializer instead
7.7
Challenge: Serialization Solution: generic lambda deserializer body of $deserializeLambda$ is an indy callsite for a cache passes lookup along to generic code that calls LMF manually tradeoff: can't fail fast for subtle serialization incompatibilties
7.8
Challenge: lock scoping lazy vals in lambdas translate into this.synchronized this used to be the anon class now it is the lambda host, which contains the lambda body in a synthetic method
7.9
Challenge: unwanted capture local methods in lambdas translate to private methods... ... in the anon class (before) ... in the lambda host class (now) lambda now retains a reference to the lambda host solution: transitive analysis to make lifted methods static
7 . 10
What's next? 2.12.0-RC1 end of the month Deliver Java 9 support incrementally Invest in benchmarking, performance of compiler/library
8
Thanks! Questions?
9
